[tahoe-lafs-trac-stream] [tahoe-lafs] #2034: Update for mutable files is sensitive to the number of servers (was: Mutable update tests are sensitive to the number of servers)
tahoe-lafs
trac at tahoe-lafs.org
Tue Jul 23 16:20:37 UTC 2013
#2034: Update for mutable files is sensitive to the number of servers
------------------------------+-----------------------
Reporter: markberger | Owner:
Type: defect | Status: new
Priority: normal | Milestone: undecided
Component: code-mutable | Version: 1.10.0
Resolution: | Keywords:
Launchpad Bug: |
------------------------------+-----------------------
Description changed by markberger:
Old description:
> When the number of servers used in allmydata.test.test_mutable.Update is
> changed from the default value of ten to something greater than ten,
> almost all of the tests fail due to key error.
>
> {{{
> ===============================================================================
> [FAIL]
> Traceback (most recent call last):
> File "/Users/markberger/Code/tahoe-
> lafs/src/allmydata/test/test_mutable.py", line 3663, in <lambda>
> self.failUnlessEqual(results, new_data))
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
> /site-packages/twisted/trial/_synctest.py", line 356, in assertEqual
> % (msg, pformat(first), pformat(second)))
> twisted.trial.unittest.FailTest: not equal:
> a = 'test datatest datatest datatest datatest datatest datatest datatest
> datatest datatest data'
> b = 'test datatest datatest datatest datatest datatest datatest datatest
> datatest datatest dataappended'
>
> allmydata.test.test_mutable.Update.test_update_sdmf
> ===============================================================================
> [ERROR]
> Traceback (most recent call last):
> Failure: allmydata.mutable.common.NotEnoughServersError: ("Publish ran
> out of good servers, last failure was: [Failure instance: Traceback:
> <type 'exceptions.KeyError'>: 0\n/Users/markberger/Code/tahoe-
> lafs/src/allmydata/mutable/publish.py:642:_push\n/Users/markberger/Code
> /tahoe-
> lafs/src/allmydata/mutable/publish.py:662:push_segment\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
> /site-
> packages/twisted/internet/defer.py:304:addCallback\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
> /site-packages/twisted/internet/defer.py:293:addCallbacks\n--- <exception
> caught here>
> ---\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
> /site-
> packages/twisted/internet/defer.py:575:_runCallbacks\n/Users/markberger/Code
> /tahoe-lafs/src/allmydata/mutable/publish.py:767:_push_segment\n]", None)
>
> allmydata.test.test_mutable.Update.test_append
> allmydata.test.test_mutable.Update.test_append_power_of_two
> allmydata.test.test_mutable.Update.test_multiple_segment_replace
> allmydata.test.test_mutable.Update.test_replace_beginning
> allmydata.test.test_mutable.Update.test_replace_segstart1
> ===============================================================================
> [ERROR]
> Traceback (most recent call last):
> Failure: allmydata.mutable.common.NotEnoughServersError: ("Publish ran
> out of good servers, last failure was: [Failure instance: Traceback:
> <type 'exceptions.KeyError'>: 1\n/Users/markberger/Code/tahoe-
> lafs/src/allmydata/mutable/publish.py:642:_push\n/Users/markberger/Code
> /tahoe-
> lafs/src/allmydata/mutable/publish.py:662:push_segment\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
> /site-
> packages/twisted/internet/defer.py:304:addCallback\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
> /site-packages/twisted/internet/defer.py:293:addCallbacks\n--- <exception
> caught here>
> ---\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
> /site-
> packages/twisted/internet/defer.py:575:_runCallbacks\n/Users/markberger/Code
> /tahoe-lafs/src/allmydata/mutable/publish.py:767:_push_segment\n]", None)
>
> allmydata.test.test_mutable.Update.test_replace_and_extend
> allmydata.test.test_mutable.Update.test_replace_in_last_segment
> allmydata.test.test_mutable.Update.test_replace_locations
> allmydata.test.test_mutable.Update.test_replace_middle
> allmydata.test.test_mutable.Update.test_replace_zero_length_middle
> ===============================================================================
> [ERROR]
> Traceback (most recent call last):
> File "/Users/markberger/Code/tahoe-
> lafs/src/allmydata/mutable/filenode.py", line 1177, in
> _build_uploadable_and_finish
> return p.update(u, offset, segments_and_bht[2], self._version)
> File "/Users/markberger/Code/tahoe-
> lafs/src/allmydata/mutable/publish.py", line 341, in update
> self._push()
> File "/Users/markberger/Code/tahoe-
> lafs/src/allmydata/mutable/publish.py", line 642, in _push
> return self.push_segment(self._current_segment)
> File "/Users/markberger/Code/tahoe-
> lafs/src/allmydata/mutable/publish.py", line 659, in push_segment
> return self._push()
> File "/Users/markberger/Code/tahoe-
> lafs/src/allmydata/mutable/publish.py", line 645, in _push
> return self.push_everything_else()
> File "/Users/markberger/Code/tahoe-
> lafs/src/allmydata/mutable/publish.py", line 778, in push_everything_else
> self.push_blockhashes()
> File "/Users/markberger/Code/tahoe-
> lafs/src/allmydata/mutable/publish.py", line 806, in push_blockhashes
> writers = self.writers[shnum]
> exceptions.KeyError: 0
>
> allmydata.test.test_mutable.Update.test_replace_zero_length_beginning
> allmydata.test.test_mutable.Update.test_replace_zero_length_segstart1
> -------------------------------------------------------------------------------
> Ran 14 tests in 58.570s
> }}}
New description:
When the number of servers used in a grid is greater than or equal to N +
k, there is a race condition when mutable files are updated.
ServermapUpdater will hit a threshold after EPSILON servers (currently
EPSILON = k) from the permuted server list do not have any shares. This
will cause the process to finish and reject any remaining responses.
However, this can occur before the ServermapUpdater has added the server
to the servermap, causing the client to reject the server's response.
Since the server is not added to the servermap, a writer for that server
is not created. This causes a key error when the updater attempts to
retrieve the corresponding writer for each share.
This race condition can be replicated by setting the number of servers for
allmydata.test.test_mutable.Update to 13.
{{{
===============================================================================
[FAIL]
Traceback (most recent call last):
File "/Users/markberger/Code/tahoe-
lafs/src/allmydata/test/test_mutable.py", line 3663, in <lambda>
self.failUnlessEqual(results, new_data))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-packages/twisted/trial/_synctest.py", line 356, in assertEqual
% (msg, pformat(first), pformat(second)))
twisted.trial.unittest.FailTest: not equal:
a = 'test datatest datatest datatest datatest datatest datatest datatest
datatest datatest data'
b = 'test datatest datatest datatest datatest datatest datatest datatest
datatest datatest dataappended'
allmydata.test.test_mutable.Update.test_update_sdmf
===============================================================================
[ERROR]
Traceback (most recent call last):
Failure: allmydata.mutable.common.NotEnoughServersError: ("Publish ran out
of good servers, last failure was: [Failure instance: Traceback: <type
'exceptions.KeyError'>: 0\n/Users/markberger/Code/tahoe-
lafs/src/allmydata/mutable/publish.py:642:_push\n/Users/markberger/Code
/tahoe-
lafs/src/allmydata/mutable/publish.py:662:push_segment\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-
packages/twisted/internet/defer.py:304:addCallback\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-packages/twisted/internet/defer.py:293:addCallbacks\n--- <exception
caught here>
---\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
packages/twisted/internet/defer.py:575:_runCallbacks\n/Users/markberger/Code
/tahoe-lafs/src/allmydata/mutable/publish.py:767:_push_segment\n]", None)
allmydata.test.test_mutable.Update.test_append
allmydata.test.test_mutable.Update.test_append_power_of_two
allmydata.test.test_mutable.Update.test_multiple_segment_replace
allmydata.test.test_mutable.Update.test_replace_beginning
allmydata.test.test_mutable.Update.test_replace_segstart1
===============================================================================
[ERROR]
Traceback (most recent call last):
Failure: allmydata.mutable.common.NotEnoughServersError: ("Publish ran out
of good servers, last failure was: [Failure instance: Traceback: <type
'exceptions.KeyError'>: 1\n/Users/markberger/Code/tahoe-
lafs/src/allmydata/mutable/publish.py:642:_push\n/Users/markberger/Code
/tahoe-
lafs/src/allmydata/mutable/publish.py:662:push_segment\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-
packages/twisted/internet/defer.py:304:addCallback\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/site-packages/twisted/internet/defer.py:293:addCallbacks\n--- <exception
caught here>
---\n/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-
packages/twisted/internet/defer.py:575:_runCallbacks\n/Users/markberger/Code
/tahoe-lafs/src/allmydata/mutable/publish.py:767:_push_segment\n]", None)
allmydata.test.test_mutable.Update.test_replace_and_extend
allmydata.test.test_mutable.Update.test_replace_in_last_segment
allmydata.test.test_mutable.Update.test_replace_locations
allmydata.test.test_mutable.Update.test_replace_middle
allmydata.test.test_mutable.Update.test_replace_zero_length_middle
===============================================================================
[ERROR]
Traceback (most recent call last):
File "/Users/markberger/Code/tahoe-
lafs/src/allmydata/mutable/filenode.py", line 1177, in
_build_uploadable_and_finish
return p.update(u, offset, segments_and_bht[2], self._version)
File "/Users/markberger/Code/tahoe-
lafs/src/allmydata/mutable/publish.py", line 341, in update
self._push()
File "/Users/markberger/Code/tahoe-
lafs/src/allmydata/mutable/publish.py", line 642, in _push
return self.push_segment(self._current_segment)
File "/Users/markberger/Code/tahoe-
lafs/src/allmydata/mutable/publish.py", line 659, in push_segment
return self._push()
File "/Users/markberger/Code/tahoe-
lafs/src/allmydata/mutable/publish.py", line 645, in _push
return self.push_everything_else()
File "/Users/markberger/Code/tahoe-
lafs/src/allmydata/mutable/publish.py", line 778, in push_everything_else
self.push_blockhashes()
File "/Users/markberger/Code/tahoe-
lafs/src/allmydata/mutable/publish.py", line 806, in push_blockhashes
writers = self.writers[shnum]
exceptions.KeyError: 0
allmydata.test.test_mutable.Update.test_replace_zero_length_beginning
allmydata.test.test_mutable.Update.test_replace_zero_length_segstart1
-------------------------------------------------------------------------------
Ran 14 tests in 58.570s
}}}
--
--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2034#comment:2>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage
More information about the tahoe-lafs-trac-stream
mailing list