[tahoe-lafs-trac-stream] [tahoe-lafs] #1921: leasedb: NonExistentShareError: can't find [share] in `shares` table
tahoe-lafs
trac at tahoe-lafs.org
Sat May 25 22:31:56 UTC 2013
#1921: leasedb: NonExistentShareError: can't find [share] in `shares` table
------------------------------+----------------------------------------
Reporter: davidsarah | Owner: davidsarah
Type: defect | Status: new
Priority: major | Milestone: soon
Component: code-storage | Version: 1.9.2
Resolution: | Keywords: leasedb accounting-crawler
Launchpad Bug: |
------------------------------+----------------------------------------
Old description:
> The attached incident was seen when running a test with a large number of
> uploads with the OpenStack cloud backend. The most relevant part seems to
> be:
> {{{
> local#18690 23:06:18.999: storage: allocate_buckets
> lp6ibjsxe6vf6ern6v3soepjh4
> local#18691 23:06:18.999: OpenStack list objects request GET
> https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_a888b70a-4771-40c0-8403-e921454e03fd/test?format=json&prefix=shares%2Flp%2Flp6ibjsxe6vf6ern6v3soepjh4%2F
> {'User-Agent': ['Tahoe-LAFS OpenStack client'], 'X-Auth-Token':
> ['be1fea33-921b-47be-b95b-99d4cc5139ea']}
> local#18692 23:06:18.999: Starting factory
> <twisted.web.client._HTTP11ClientFactory instance at 0x4433830>
> local#18693 23:06:19.504: OpenStack list objects response: 200 OK
> local#18694 23:06:19.505: OpenStack list read 201 bytes, parsed as 1
> items
> local#18695 23:06:19.505: OpenStack get object request GET
> https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_a888b70a-4771-40c0-8403-e921454e03fd/test/shares/lp/lpcjuif2ixx6khivy6zxdmvofe/0
> {'User-Agent': ['Tahoe-LAFS OpenStack client'], 'X-Auth-Token':
> ['be1fea33-921b-47be-b95b-99d4cc5139ea']}
> local#18696 23:06:19.506: Starting factory
> <twisted.web.client._HTTP11ClientFactory instance at 0x442b320>
> local#18697 23:06:19.506: Stopping factory
> <twisted.web.client._HTTP11ClientFactory instance at 0x46b6c68>
> local#18698 23:06:19.520: OpenStack list objects response: 200 OK
> local#18699 23:06:19.521: OpenStack list read 2 bytes, parsed as 0 items
> local#18700 23:06:19.523: Stopping factory
> <twisted.web.client._HTTP11ClientFactory instance at 0x4433830>
> local#18701 23:06:19.537: OpenStack put object request PUT
> https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_a888b70a-4771-40c0-8403-e921454e03fd/test/shares/lp/lp6ibjsxe6vf6ern6v3soepjh4/0
> {'User-Agent': ['Tahoe-LAFS OpenStack client'], 'Content-Type':
> ['application/octet-stream'], 'X-Auth-Token': ['be1fea33-921b-47be-b95b-
> 99d4cc5139ea']}
> local#18702 23:06:19.537: Starting factory
> <twisted.web.client._HTTP11ClientFactory instance at 0x433fef0>
> local#18703 23:06:20.020: OpenStack get object response: 200 OK
> local#18704 23:06:20.022: share SI=lp6ibjsxe6vf6ern6v3soepjh4 shnum=0
> unexpectedly disappeared [INCIDENT-TRIGGER]
> local#18705 23:06:20.059: Stopping factory
> <twisted.web.client._HTTP11ClientFactory instance at 0x442b320>
> local#18706 23:06:20.062: OpenStack list objects request GET
> https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_a888b70a-4771-40c0-8403-e921454e03fd/test?format=json&prefix=shares%2Flq%2F
> {'User-Agent': ['Tahoe-LAFS OpenStack client'], 'X-Auth-Token':
> ['be1fea33-921b-47be-b95b-99d4cc5139ea']}
> local#18707 23:06:20.062: Starting factory
> <twisted.web.client._HTTP11ClientFactory instance at 0x4341e60>
> local#18708 23:06:20.521: OpenStack put object response: 201 Created
> local#18709 23:06:20.527: an inbound callRemote that we [n4zt] executed
> (on behalf of someone else, TubID uzie) failed
> local#18710 23:06:20.527: reqID=6970,
> rref=<allmydata.storage.bucket.BucketWriter object at 0x31a9050>,
> methname=RIBucketWriter.close
> local#18711 23:06:20.527: args=[]
> local#18712 23:06:20.527: kwargs={}
> local#18713 23:06:20.527: the LOCAL failure was:
> FAILURE:
> [CopiedFailure instance: Traceback from remote host -- Traceback (most
> recent call last):
> File "/usr/lib/python2.7/dist-packages/foolscap/eventual.py", line 26,
> in _turn
> cb(*args, **kwargs)
> File "/home/davidsarah/tahoe/bitsys/allmydata-
> tahoe-1.9.0.post363/src/allmydata/util/deferredutil.py", line 55, in
> _with_log
> op(res)
> File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py",
> line 368, in callback
> self._startRunCallbacks(result)
> File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py",
> line 464, in _startRunCallbacks
> self._runCallbacks()
> --- <exception caught here> ---
> File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py",
> line 551, in _runCallbacks
> current.result = callback(current.result, *args, **kw)
> File "/home/davidsarah/tahoe/bitsys/allmydata-
> tahoe-1.9.0.post363/src/allmydata/storage/bucket.py", line 62, in
> _got_used_space
> self._account.add_or_renew_default_lease(storage_index, shnum)
> File "/home/davidsarah/tahoe/bitsys/allmydata-
> tahoe-1.9.0.post363/src/allmydata/storage/account.py", line 58, in
> add_or_renew_default_lease
> return self.add_or_renew_lease(storage_index, shnum, renewal_time,
> expiration_time)
> File "/home/davidsarah/tahoe/bitsys/allmydata-
> tahoe-1.9.0.post363/src/allmydata/storage/account.py", line 63, in
> add_or_renew_lease
> renewal_time, expiration_time)
> File "/home/davidsarah/tahoe/bitsys/allmydata-
> tahoe-1.9.0.post363/src/allmydata/storage/leasedb.py", line 250, in
> add_or_renew_leases
> raise NonExistentShareError(si_s, shnum)
> allmydata.storage.leasedb.NonExistentShareError: can't find
> SI='lp6ibjsxe6vf6ern6v3soepjh4' shnum=0 in `shares` table
> ]
> local#18714 23:06:20.533: Stopping factory
> <twisted.web.client._HTTP11ClientFactory instance at 0x433fef0>
> local#18715 23:06:20.545: storage: aborting write to share None
> }}}
> I'll avoid speculation in the ticket description, but I don't think this
> is specific to OpenStack.
New description:
The attached incident was seen when running a test with a large number of
uploads with the OpenStack cloud backend. The most relevant part seems to
be:
{{{
local#18690 23:06:18.999: storage: allocate_buckets
lp6ibjsxe6vf6ern6v3soepjh4
local#18691 23:06:18.999: OpenStack list objects request GET
https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_a888b70a-4771-40c0-8403-e921454e03fd/test?format=json&prefix=shares%2Flp%2Flp6ibjsxe6vf6ern6v3soepjh4%2F
{'User-Agent': ['Tahoe-LAFS OpenStack client'], 'X-Auth-Token': ['be1fea33
-921b-47be-b95b-99d4cc5139ea']}
local#18692 23:06:18.999: Starting factory
<twisted.web.client._HTTP11ClientFactory instance at 0x4433830>
local#18693 23:06:19.504: OpenStack list objects response: 200 OK
local#18694 23:06:19.505: OpenStack list read 201 bytes, parsed as 1 items
local#18695 23:06:19.505: OpenStack get object request GET
https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_a888b70a-4771-40c0-8403-e921454e03fd/test/shares/lp/lpcjuif2ixx6khivy6zxdmvofe/0
{'User-Agent': ['Tahoe-LAFS OpenStack client'], 'X-Auth-Token': ['be1fea33
-921b-47be-b95b-99d4cc5139ea']}
local#18696 23:06:19.506: Starting factory
<twisted.web.client._HTTP11ClientFactory instance at 0x442b320>
local#18697 23:06:19.506: Stopping factory
<twisted.web.client._HTTP11ClientFactory instance at 0x46b6c68>
local#18698 23:06:19.520: OpenStack list objects response: 200 OK
local#18699 23:06:19.521: OpenStack list read 2 bytes, parsed as 0 items
local#18700 23:06:19.523: Stopping factory
<twisted.web.client._HTTP11ClientFactory instance at 0x4433830>
local#18701 23:06:19.537: OpenStack put object request PUT
https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_a888b70a-4771-40c0-8403-e921454e03fd/test/shares/lp/lp6ibjsxe6vf6ern6v3soepjh4/0
{'User-Agent': ['Tahoe-LAFS OpenStack client'], 'Content-Type':
['application/octet-stream'], 'X-Auth-Token': ['be1fea33-921b-47be-b95b-
99d4cc5139ea']}
local#18702 23:06:19.537: Starting factory
<twisted.web.client._HTTP11ClientFactory instance at 0x433fef0>
local#18703 23:06:20.020: OpenStack get object response: 200 OK
local#18704 23:06:20.022: share SI=lp6ibjsxe6vf6ern6v3soepjh4 shnum=0
unexpectedly disappeared [INCIDENT-TRIGGER]
local#18705 23:06:20.059: Stopping factory
<twisted.web.client._HTTP11ClientFactory instance at 0x442b320>
local#18706 23:06:20.062: OpenStack list objects request GET
https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_a888b70a-4771-40c0-8403-e921454e03fd/test?format=json&prefix=shares%2Flq%2F
{'User-Agent': ['Tahoe-LAFS OpenStack client'], 'X-Auth-Token': ['be1fea33
-921b-47be-b95b-99d4cc5139ea']}
local#18707 23:06:20.062: Starting factory
<twisted.web.client._HTTP11ClientFactory instance at 0x4341e60>
local#18708 23:06:20.521: OpenStack put object response: 201 Created
local#18709 23:06:20.527: an inbound callRemote that we [n4zt] executed
(on behalf of someone else, TubID uzie) failed
local#18710 23:06:20.527: reqID=6970,
rref=<allmydata.storage.bucket.BucketWriter object at 0x31a9050>,
methname=RIBucketWriter.close
local#18711 23:06:20.527: args=[]
local#18712 23:06:20.527: kwargs={}
local#18713 23:06:20.527: the LOCAL failure was:
FAILURE:
[CopiedFailure instance: Traceback from remote host -- Traceback (most
recent call last):
File "/usr/lib/python2.7/dist-packages/foolscap/eventual.py", line 26,
in _turn
cb(*args, **kwargs)
File "/home/davidsarah/tahoe/bitsys/allmydata-
tahoe-1.9.0.post363/src/allmydata/util/deferredutil.py", line 55, in
_with_log
op(res)
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
368, in callback
self._startRunCallbacks(result)
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
464, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
551, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/home/davidsarah/tahoe/bitsys/allmydata-
tahoe-1.9.0.post363/src/allmydata/storage/bucket.py", line 62, in
_got_used_space
self._account.add_or_renew_default_lease(storage_index, shnum)
File "/home/davidsarah/tahoe/bitsys/allmydata-
tahoe-1.9.0.post363/src/allmydata/storage/account.py", line 58, in
add_or_renew_default_lease
return self.add_or_renew_lease(storage_index, shnum, renewal_time,
expiration_time)
File "/home/davidsarah/tahoe/bitsys/allmydata-
tahoe-1.9.0.post363/src/allmydata/storage/account.py", line 63, in
add_or_renew_lease
renewal_time, expiration_time)
File "/home/davidsarah/tahoe/bitsys/allmydata-
tahoe-1.9.0.post363/src/allmydata/storage/leasedb.py", line 250, in
add_or_renew_leases
raise NonExistentShareError(si_s, shnum)
allmydata.storage.leasedb.NonExistentShareError: can't find
SI='lp6ibjsxe6vf6ern6v3soepjh4' shnum=0 in `shares` table
]
local#18714 23:06:20.533: Stopping factory
<twisted.web.client._HTTP11ClientFactory instance at 0x433fef0>
local#18715 23:06:20.545: storage: aborting write to share None
}}}
I'll avoid speculation in the ticket description, but I don't think this
is specific to OpenStack.
--
Comment (by daira):
#1987 may be the same bug as this. I'm not marking them as duplicates
because I'm not sure of that yet.
--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1921#comment:6>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage
More information about the tahoe-lafs-trac-stream
mailing list