[tahoe-lafs-trac-stream] [Tahoe-LAFS] #812: server-side crawlers: tolerate corrupted shares, verify shares

[Tahoe-LAFS]

#812: server-side crawlers: tolerate corrupted shares, verify shares
------------------------------+-------------------------
     Reporter:  zooko         |      Owner:  warner
         Type:  defect        |     Status:  new
     Priority:  major         |  Milestone:  undecided
    Component:  code-storage  |    Version:  1.4.1
   Resolution:                |   Keywords:  reliability
Launchpad Bug:                |
------------------------------+-------------------------

Comment (by zooko):

 My consulting client (codename "WAG") (see comment:6:ticket:1278) has
 corrupt shares. This message is in the storage server's twistd.log:

 {{{
 2014-10-16 20:28:32+0530 [-] lease-checker error processing /GFS2_tahoe
 /.tahoe-filestore1/storage/shares/jz/jze7ynr5mtrehxj7vchc3dqtpa/5
 2014-10-16 20:28:32+0530 [-] Unhandled Error
         Traceback (most recent call last):
           File "/data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-x86_64.egg/twisted/internet/base.py",
 line 824, in runUntilCurrent
             call.func(*call.args, **call.kw)
           File "/data/allmydata-
 tahoe-1.10.0/src/allmydata/storage/crawler.py", line 261, in start_slice
             self.start_current_prefix(start_slice)
           File "/data/allmydata-
 tahoe-1.10.0/src/allmydata/storage/crawler.py", line 320, in
 start_current_prefix
             buckets, start_slice)
           File "/data/allmydata-
 tahoe-1.10.0/src/allmydata/storage/crawler.py", line 360, in
 process_prefixdir
             self.process_bucket(cycle, prefix, prefixdir, bucket)
         --- <exception caught here> ---
           File "/data/allmydata-
 tahoe-1.10.0/src/allmydata/storage/expirer.py", line 128, in
 process_bucket
             wks = self.process_share(sharefile)
           File "/data/allmydata-
 tahoe-1.10.0/src/allmydata/storage/expirer.py", line 171, in process_share
             for li in sf.get_leases():
           File "/data/allmydata-
 tahoe-1.10.0/src/allmydata/storage/mutable.py", line 265, in get_leases
             for i, lease in self._enumerate_leases(f):
           File "/data/allmydata-
 tahoe-1.10.0/src/allmydata/storage/mutable.py", line 270, in
 _enumerate_leases
             for i in range(self._get_num_lease_slots(f)):
           File "/data/allmydata-
 tahoe-1.10.0/src/allmydata/storage/mutable.py", line 250, in
 _get_num_lease_slots
             num_extra_leases = self._read_num_extra_leases(f)
           File "/data/allmydata-
 tahoe-1.10.0/src/allmydata/storage/mutable.py", line 133, in
 _read_num_extra_leases
             (num_extra_leases,) = struct.unpack(">L", f.read(4))
         struct.error: unpack requires a string argument of length 4

 }}}

 And this message is in an incident report file generated by the node
 (which is both the storage server and the gateway):

 {{{
 CorruptShareError server=5e4d7u5c shnum[2]: block hash tree failure: new
 hash soakg4py7uswum3y6vzvyk4qt3736ia37l7ti22mce3ntesuhe7a does not match
 existing hash wuoq2uiuyzope6gwyvucwx3b5q47hn46kvynkhsvzh7jndrcowla at [2
 of 3] (leaf [1] of 2)
 /data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-
 x86_64.egg/twisted/internet/defer.py:575:_runCallbacks
 /data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-
 x86_64.egg/twisted/internet/defer.py:827:_cbDeferred
 /data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-
 x86_64.egg/twisted/internet/defer.py:380:callback
 /data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-
 x86_64.egg/twisted/internet/defer.py:488:_startRunCallbacks
 --- <exception caught here> ---
 /data/allmydata-tahoe-1.10.0/support/lib/python2.6/site-
 packages/Twisted-13.0.0-py2.6-linux-
 x86_64.egg/twisted/internet/defer.py:575:_runCallbacks
 /data/allmydata-
 tahoe-1.10.0/src/allmydata/mutable/retrieve.py:797:_validate_block
 }}}

 This is with this version of Tahoe-LAFS:

 {{{
 Nevow: 0.11.1
 Twisted: 13.0.0
 allmydata-tahoe: 1.10.0
 foolscap: 0.6.4
 mock: 1.0.1
 platform: Linux-redhat_6.5-x86_64-64bit_ELF
 pyOpenSSL: 0.10
 pyasn1: 0.1.7
 pycrypto: 2.6.1
 pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958
 python: 2.6.6
 setuptools: 0.6c16dev4
 simplejson: 2.0.9
 twisted: 13.0.0
 zfec: 1.4.7
 zope.interface: unknown
 }}}

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/812#comment:7>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage