[tahoe-lafs-trac-stream] [tahoe-lafs] #1803: S3 backend: AttributeError: 'NoneType' object has no attribute 'startswith'
tahoe-lafs
trac at tahoe-lafs.org
Thu Nov 22 01:25:50 UTC 2012
#1803: S3 backend: AttributeError: 'NoneType' object has no attribute 'startswith'
------------------------------+------------------------------------
Reporter: zooko | Owner: davidsarah
Type: defect | Status: assigned
Priority: major | Milestone: undecided
Component: code-storage | Version: 1.9.0-s3branch
Resolution: | Keywords: s3-backend error txaws
Launchpad Bug: |
------------------------------+------------------------------------
Changes (by davidsarah):
* keywords: s3-backend error => s3-backend error txaws
Old description:
> This was after it had been doing a "tahoe backup" job for about 90
> minutes. There could have been a transient network failure.
>
> {{{
> File "/home/zooko/playground/tahoe-lafs/cloud-
> backend/src/allmydata/scripts/tahoe_backup.py", line 305, in upload
> raise HTTPError("Error during file PUT", resp)
> HTTPError: Error during file PUT: 500 Internal Server Error
> "Traceback (most recent call last):\x0a File \"/usr/local/lib/python2.7
> /dist-packages/foolscap-0.6.3.post0-py2.7.egg/foolscap/call.py\", line
> 753, in receiveClose\x0a self.request.fail(f)\x0a File
> \"/usr/local/lib/python2.7/dist-
> packages/foolscap-0.6.3.post0-py2.7.egg/foolscap/call.py\", line 95, in
> fail\x0a self.deferred.errback(why)\x0a File \"/home/zooko/playground
> /tahoe-lafs/cloud-backend/support/lib/python2.7/site-
> packages/Twisted-11.1.0-py2.7-linux-
> x86_64.egg/twisted/internet/defer.py\", line 391, in errback\x0a
> self._startRunCallbacks(fail)\x0a File \"/home/zooko/playground/tahoe-
> lafs/cloud-backend/support/lib/python2.7/site-
> packages/Twisted-11.1.0-py2.7-linux-
> x86_64.egg/twisted/internet/defer.py\", line 458, in
> _startRunCallbacks\x0a self._runCallbacks()\x0a--- <exception caught
> here> ---\x0a File \"/home/zooko/playground/tahoe-lafs/cloud-
> backend/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-
> x86_64.egg/twisted/internet/defer.py\", line 545, in _runCallbacks\x0a
> current.result = callback(current.result, *args, **kw)\x0a File
> \"/home/zooko/playground/tahoe-lafs/cloud-
> backend/src/allmydata/immutable/upload.py\", line 604, in
> _got_response\x0a return self._loop()\x0a File
> \"/home/zooko/playground/tahoe-lafs/cloud-
> backend/src/allmydata/immutable/upload.py\", line 516, in _loop\x0a
> return self._failed(msg)\x0a File \"/home/zooko/playground/tahoe-lafs
> /cloud-backend/src/allmydata/immutable/upload.py\", line 617, in
> _failed\x0a raise
> UploadUnhappinessError(msg)\x0aallmydata.interfaces.UploadUnhappinessError:
> server selection failed for <Tahoe2ServerSelector for upload yyoj4>:
> shares could be placed or found on only 0 server(s). We were asked to
> place shares on at least 1 server(s) such that any 1 of them have enough
> shares to recover the file. (placed 0 shares out of 1 total (1 homeless),
> want to place shares on at least 1 servers such that any 1 of them have
> enough shares to recover the file, sent 1 queries to 1 servers, 0 queries
> placed some shares, 1 placed none (of which 0 placed none due to the
> server being full and 1 placed none due to an error)) (last failure (from
> <ServerTracker for server 66rayr and SI yyoj4>) was: [Failure instance:
> Traceback (failure with no frames): <class
> 'foolscap.tokens.RemoteException'>: <RemoteException around
> '[CopiedFailure instance: Traceback from remote host -- Traceback (most
> recent call last):\x0a File \"/usr/local/lib/python2.6/dist-
> packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/tcp.py\",
> line 277, in connectionLost\x0a protocol.connectionLost(reason)\x0a
> File \"/usr/local/lib/python2.6/dist-
> packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/web/client.py\",
> line 191, in connectionLost\x0a
> self.factory._disconnectedDeferred.callback(None)\x0a File
> \"/usr/local/lib/python2.6/dist-
> packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
> line 362, in callback\x0a self._startRunCallbacks(result)\x0a File
> \"/usr/local/lib/python2.6/dist-
> packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
> line 458, in _startRunCallbacks\x0a self._runCallbacks()\x0a---
> <exception caught here> ---\x0a File \"/usr/local/lib/python2.6/dist-
> packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
> line 545, in _runCallbacks\x0a current.result =
> callback(current.result, *args, **kw)\x0a File
> \"/home/customer/LAFS_source/src/allmydata/storage/backends/s3/s3_backend.py\",
> line 20, in _make_share\x0a if
> data.startswith(MUTABLE_MAGIC):\x0aexceptions.AttributeError: 'NoneType'
> object has no attribute 'startswith'\x0a]'>\x0a])\x0a"
> }}}
>
> The client is actually the cloud backend, not trunk.
>
> {{{
> allmydata-tahoe: 1.9.0-r5930
> foolscap: 0.6.3.post0
> pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958
> zfec: 1.4.24
> Twisted: 11.1.0
> Nevow: 0.10.0
> zope.interface: unknown
> python: 2.7.3
> platform: Linux-Ubuntu_12.04-x86_64-64bit_ELF
> pyOpenSSL: 0.12
> simplejson: 2.3.2
> pycrypto: 2.5
> pyasn1: unknown
> mock: 0.8.0beta3
> txAWS: 0.2.1.post4
> Epsilon: 0.6.0
> setuptools: 0.6c16dev3
> }}}
>
> The storage server is LAE's ticket999-S3-backend branch.
New description:
This was after it had been doing a "tahoe backup" job for about 90
minutes. There could have been a transient network failure.
{{{
File "/home/zooko/playground/tahoe-lafs/cloud-
backend/src/allmydata/scripts/tahoe_backup.py", line 305, in upload
raise HTTPError("Error during file PUT", resp)
HTTPError: Error during file PUT: 500 Internal Server Error
"Traceback (most recent call last):\x0a File \"/usr/local/lib/python2.7
/dist-packages/foolscap-0.6.3.post0-py2.7.egg/foolscap/call.py\", line
753, in receiveClose\x0a self.request.fail(f)\x0a File
\"/usr/local/lib/python2.7/dist-
packages/foolscap-0.6.3.post0-py2.7.egg/foolscap/call.py\", line 95, in
fail\x0a self.deferred.errback(why)\x0a File \"/home/zooko/playground
/tahoe-lafs/cloud-backend/support/lib/python2.7/site-
packages/Twisted-11.1.0-py2.7-linux-
x86_64.egg/twisted/internet/defer.py\", line 391, in errback\x0a
self._startRunCallbacks(fail)\x0a File \"/home/zooko/playground/tahoe-
lafs/cloud-backend/support/lib/python2.7/site-
packages/Twisted-11.1.0-py2.7-linux-
x86_64.egg/twisted/internet/defer.py\", line 458, in
_startRunCallbacks\x0a self._runCallbacks()\x0a--- <exception caught
here> ---\x0a File \"/home/zooko/playground/tahoe-lafs/cloud-
backend/support/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-linux-
x86_64.egg/twisted/internet/defer.py\", line 545, in _runCallbacks\x0a
current.result = callback(current.result, *args, **kw)\x0a File
\"/home/zooko/playground/tahoe-lafs/cloud-
backend/src/allmydata/immutable/upload.py\", line 604, in
_got_response\x0a return self._loop()\x0a File \"/home/zooko/playground
/tahoe-lafs/cloud-backend/src/allmydata/immutable/upload.py\", line 516,
in _loop\x0a return self._failed(msg)\x0a File
\"/home/zooko/playground/tahoe-lafs/cloud-
backend/src/allmydata/immutable/upload.py\", line 617, in _failed\x0a
raise
UploadUnhappinessError(msg)\x0aallmydata.interfaces.UploadUnhappinessError:
server selection failed for <Tahoe2ServerSelector for upload yyoj4>:
shares could be placed or found on only 0 server(s). We were asked to
place shares on at least 1 server(s) such that any 1 of them have enough
shares to recover the file. (placed 0 shares out of 1 total (1 homeless),
want to place shares on at least 1 servers such that any 1 of them have
enough shares to recover the file, sent 1 queries to 1 servers, 0 queries
placed some shares, 1 placed none (of which 0 placed none due to the
server being full and 1 placed none due to an error)) (last failure (from
<ServerTracker for server 66rayr and SI yyoj4>) was: [Failure instance:
Traceback (failure with no frames): <class
'foolscap.tokens.RemoteException'>: <RemoteException around
'[CopiedFailure instance: Traceback from remote host -- Traceback (most
recent call last):\x0a File \"/usr/local/lib/python2.6/dist-
packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/tcp.py\",
line 277, in connectionLost\x0a protocol.connectionLost(reason)\x0a
File \"/usr/local/lib/python2.6/dist-
packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/web/client.py\", line
191, in connectionLost\x0a
self.factory._disconnectedDeferred.callback(None)\x0a File
\"/usr/local/lib/python2.6/dist-
packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
line 362, in callback\x0a self._startRunCallbacks(result)\x0a File
\"/usr/local/lib/python2.6/dist-
packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
line 458, in _startRunCallbacks\x0a self._runCallbacks()\x0a---
<exception caught here> ---\x0a File \"/usr/local/lib/python2.6/dist-
packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
line 545, in _runCallbacks\x0a current.result =
callback(current.result, *args, **kw)\x0a File
\"/home/customer/LAFS_source/src/allmydata/storage/backends/s3/s3_backend.py\",
line 20, in _make_share\x0a if
data.startswith(MUTABLE_MAGIC):\x0aexceptions.AttributeError: 'NoneType'
object has no attribute 'startswith'\x0a]'>\x0a])\x0a"
}}}
The client is actually the cloud backend, not trunk.
{{{
allmydata-tahoe: 1.9.0-r5930
foolscap: 0.6.3.post0
pycryptopp: 0.6.0.1206569328141510525648634803928199668821045408958
zfec: 1.4.24
Twisted: 11.1.0
Nevow: 0.10.0
zope.interface: unknown
python: 2.7.3
platform: Linux-Ubuntu_12.04-x86_64-64bit_ELF
pyOpenSSL: 0.12
simplejson: 2.3.2
pycrypto: 2.5
pyasn1: unknown
mock: 0.8.0beta3
txAWS: 0.2.1.post4
Epsilon: 0.6.0
setuptools: 0.6c16dev3
}}}
The storage server is LAE's ticket999-S3-backend branch.
--
--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1803#comment:3>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage
More information about the tahoe-lafs-trac-stream
mailing list