[tahoe-lafs-trac-stream] [tahoe-lafs] #1920: cloud backend: a failed upload can leave chunk objects that prevent subsequent uploads of the same file

[tahoe-lafs]

#1920: cloud backend: a failed upload can leave chunk objects that prevent
subsequent uploads of the same file
-------------------------+-------------------------------------------------
     Reporter:           |      Owner:  davidsarah
  davidsarah             |     Status:  assigned
         Type:  defect   |  Milestone:  undecided
     Priority:  normal   |    Version:  1.9.2
    Component:  code-    |   Keywords:  cloud-backend openstack upload
  storage                |  reliability
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------
Changes (by davidsarah):

 * status:  new => assigned


Old description:

> When attempting to upload a ~10 MiB file using the OpenStack cloud
> backend (to a Rackspace account), the upload failed due to a DNS error:
> {{{
> <class 'allmydata.interfaces.UploadUnhappinessError'>:
> shares could be placed or found on only 0 server(s). We were asked to
> place
> shares on at least 1 server(s) such that any 1 of them have enough shares
> to
> recover the file.:
> [Failure instance: Traceback (failure with no frames):
> <class 'allmydata.util.pipeline.PipelineError'>:
> <PipelineError error=([Failure instance: Traceback (failure with no
> frames):
> <class 'foolscap.tokens.RemoteException'>:
> <RemoteException around '[CopiedFailure instance: Traceback from remote
> host --
> Traceback (most recent call last):
> Failure: twisted.internet.error.DNSLookupError: DNS lookup failed:
> address
> 'storage101.dfw1.clouddrive.com' not found: [Errno -2] Name or service
> not known.
> ]'> ])> ]
> }}}
>
> That error is not itself the subject of this ticket. The issue for this
> ticket is that subsequent uploads of the same immutable file also failed,
> even after the DNS error had resolved itself:
> {{{
> <class 'allmydata.interfaces.UploadUnhappinessError'>:
> server selection failed for <Tahoe2ServerSelector for upload sefes>:
> shares could be placed or found on only 0 server(s). We were asked to
> place
> shares on at least 1 server(s) such that any 1 of them have enough shares
> to
> recover the file. (placed 0 shares out of 1 total (1 homeless), want to
> place
> shares on at least 1 servers such that any 1 of them have enough shares
> to
> recover the file, sent 1 queries to 1 servers, 0 queries placed some
> shares,
> 1 placed none (of which 0 placed none due to the server being full and 1
> placed
> none due to an error)) (last failure (from <ServerTracker for server
> kdu2jtww
> and SI sefes>) was:
> [Failure instance: Traceback (failure with no frames):
> <class 'foolscap.tokens.RemoteException'>:
> <RemoteException around '[CopiedFailure instance: Traceback from remote
> host --
> Traceback (most recent call last):
> [...]
> File
> "/home/davidsarah/tahoe/git/working/src/allmydata/storage/backends/cloud/cloud_common.py",
> line 339, in _retry
>   d2 = self._handle_error(f, 1, None, description, operation, *args,
> **kwargs)
> File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
> 551, in _runCallbacks
>   current.result = callback(current.result, *args, **kw)
> File
> "/home/davidsarah/tahoe/git/working/src/allmydata/storage/backends/cloud/openstack/openstack_container.py",
> line 81, in _got_response
>   message="unexpected response code %r %s" % (response.code,
> response.phrase))
> allmydata.storage.backends.cloud.cloud_common.CloudError:
> ("try 1 failed: GET object ('shares/se/sefeslgzc4su3i66b72aytmebm/0',)
> {}", 404,
> 'unexpected response code 404 Not Found', None) ]'> ])
> }}}
>
> On looking at the container contents via the Rackspace Cloud Files WUI,
> there is only one chunk object stored for this file, with key:
> {{{
> shares/se/sefeslgzc4su3i66b72aytmebm/0.5
> }}}

New description:

 When attempting to upload a ~10 MiB file using the !OpenStack cloud
 backend (to a Rackspace account), the upload failed due to a DNS error:
 {{{
 <class 'allmydata.interfaces.UploadUnhappinessError'>:
 shares could be placed or found on only 0 server(s). We were asked to
 place
 shares on at least 1 server(s) such that any 1 of them have enough shares
 to
 recover the file.:
 [Failure instance: Traceback (failure with no frames):
 <class 'allmydata.util.pipeline.PipelineError'>:
 <PipelineError error=([Failure instance: Traceback (failure with no
 frames):
 <class 'foolscap.tokens.RemoteException'>:
 <RemoteException around '[CopiedFailure instance: Traceback from remote
 host --
 Traceback (most recent call last):
 Failure: twisted.internet.error.DNSLookupError: DNS lookup failed: address
 'storage101.dfw1.clouddrive.com' not found: [Errno -2] Name or service not
 known.
 ]'> ])> ]
 }}}

 That error is not itself the subject of this ticket. The issue for this
 ticket is that subsequent uploads of the same immutable file also failed,
 even after the DNS error had resolved itself:
 {{{
 <class 'allmydata.interfaces.UploadUnhappinessError'>:
 server selection failed for <Tahoe2ServerSelector for upload sefes>:
 shares could be placed or found on only 0 server(s). We were asked to
 place
 shares on at least 1 server(s) such that any 1 of them have enough shares
 to
 recover the file. (placed 0 shares out of 1 total (1 homeless), want to
 place
 shares on at least 1 servers such that any 1 of them have enough shares to
 recover the file, sent 1 queries to 1 servers, 0 queries placed some
 shares,
 1 placed none (of which 0 placed none due to the server being full and 1
 placed
 none due to an error)) (last failure (from <ServerTracker for server
 kdu2jtww
 and SI sefes>) was:
 [Failure instance: Traceback (failure with no frames):
 <class 'foolscap.tokens.RemoteException'>:
 <RemoteException around '[CopiedFailure instance: Traceback from remote
 host --
 Traceback (most recent call last):
 [...]
 File
 "/home/davidsarah/tahoe/git/working/src/allmydata/storage/backends/cloud/cloud_common.py",
 line 339, in _retry
   d2 = self._handle_error(f, 1, None, description, operation, *args,
 **kwargs)
 File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
 551, in _runCallbacks
   current.result = callback(current.result, *args, **kw)
 File
 "/home/davidsarah/tahoe/git/working/src/allmydata/storage/backends/cloud/openstack/openstack_container.py",
 line 81, in _got_response
   message="unexpected response code %r %s" % (response.code,
 response.phrase))
 allmydata.storage.backends.cloud.cloud_common.CloudError:
 ("try 1 failed: GET object ('shares/se/sefeslgzc4su3i66b72aytmebm/0',)
 {}", 404,
 'unexpected response code 404 Not Found', None) ]'> ])
 }}}

 On looking at the container contents via the Rackspace Cloud Files WUI,
 there is only one chunk object stored for this file, with key:
 {{{
 shares/se/sefeslgzc4su3i66b72aytmebm/0.5
 }}}
 (i.e. the 6th chunk).

--

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1920#comment:2>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage