[tahoe-lafs-trac-stream] [tahoe-lafs] #1670: KeyError in mutable read-modify-write
tahoe-lafs
trac at tahoe-lafs.org
Mon Jun 17 23:22:56 UTC 2013
#1670: KeyError in mutable read-modify-write
-------------------------+-------------------------------------------------
Reporter: vikarti | Owner: zooko
Type: defect | Status: assigned
Priority: normal | Milestone: 1.11.0
Component: code- | Version: 1.9.1
mutable | Keywords: mutable retrieve error tahoe-backup
Resolution: | regression test-needed
Launchpad Bug: |
-------------------------+-------------------------------------------------
Old description:
> Mac OS X 10.7 gateway,
>
> mutable.format is set to mdmf
>
> sometimes tahoe backup process several files fine and later stops
> if it stops - it's always same place
> like this
> {{{
> skipping "/Users/vikarti/Calibre Library/Zooko Wilcox-O'Hearn_ Brian
> Warner/Tahoe - The Least-Authority Filesystem (1347)/Tahoe - The Least-
> Authority Filesystem - Zooko Wilcox-O'Hearn_ Brian Warner.pdf"..
> re-using old directory for "/Users/vikarti/Calibre Library/Zooko
> Wilcox-O'Hearn_ Brian Warner/Tahoe - The Least-Authority Filesystem
> (1347)"
> re-using old directory for "/Users/vikarti/Calibre Library/Zooko
> Wilcox-O'Hearn_ Brian Warner"
> re-using old directory for '/Users/vikarti/Calibre Library'
> Traceback (most recent call last):
> File "/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/support/bin/tahoe", line 9, in <module>
> load_entry_point('allmydata-tahoe==1.9.1', 'console_scripts',
> 'tahoe')()
> File "/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/scripts/runner.py", line 113, in run
> rc = runner(sys.argv[1:], install_node_control=install_node_control)
> File "/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/scripts/runner.py", line 99, in runner
> rc = cli.dispatch[command](so)
> File "/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/scripts/cli.py", line 569, in backup
> rc = tahoe_backup.backup(options)
> File "/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py", line 325, in backup
> return bu.run()
> File "/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py", line 123, in run
> put_child(archives_url, now, new_backup_dircap)
> File "/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py", line 57, in put_child
> raise HTTPError("Error during put_child", resp)
> allmydata.scripts.common_http.HTTPError: Error during put_child: 500
> Internal Server Error
> "Traceback (most recent call last):
> File
> \"/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/base.py\",
> line 793, in runUntilCurrent
> call.func(*call.args, **call.kw)
> File \"/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/support/lib/python2.7/site-
> packages/foolscap-0.6.3-py2.7.egg/foolscap/eventual.py\", line 26, in
> _turn
> cb(*args, **kwargs)\x0a File
> \"/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py\",
> line 361, in callback
> self._startRunCallbacks(result)\x0a File
> \"/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py\",
> line 455, in _startRunCallbacks
> self._runCallbacks()
> --- <exception caught here> ---
> File
> \"/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py\",
> line 542, in _runCallbacks
> current.result = callback(current.result, *args, **kw)
> File \"/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/mutable/filenode.py\", line 855, in
> <lambda>\x0a self._modify_once(modifier, first_time))\x0a File
> \"/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/mutable/filenode.py\", line 881, in
> _modify_once
> d = self._try_to_download_data()
> File \"/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/mutable/filenode.py\", line 959, in
> _try_to_download_data
> d = self._read(c, fetch_privkey=True)
> File \"/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/mutable/filenode.py\", line 980, in _read
> d = r.download(consumer, offset, size)
> File \"/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/mutable/retrieve.py\", line 237, in download
> self._setup_download()
> File \"/Users/vikarti/tahoe-new/allmydata-
> tahoe-1.9.1/src/allmydata/mutable/retrieve.py\", line 277, in
> _setup_download
> shares = versionmap[self.verinfo]\x0aexceptions.KeyError: (5,
> '\\x11c\\x1c\\xf7\\xc9\\xae1e\\xa2\\xdaF\\xaf\\xb6S\\x03IH\\x84\\x11#\\xb5\\xf2TS\\xfb\\xde\\x08\\x81\\x01\\xf0\\xd3\\xec',
> None, 131073, 1080, 3, 10,
> '\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x05\\x11c\\x1c\\xf7\\xc9\\xae1e\\xa2\\xdaF\\xaf\\xb6S\\x03IH\\x84\\x11#\\xb5\\xf2TS\\xfb\\xde\\x08\\x81\\x01\\xf0\\xd3\\xec\\x03\\n\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x048',
> (('EOF', 2575), ('verification_key', 1729), ('enc_privkey', 123),
> ('share_data', 2167), ('signature', 1473), ('block_hash_tree', 2543),
> ('share_hash_chain', 1337), ('verification_key_end', 2021)))
> "
> }}}
New description:
Mac OS X 10.7 gateway,
mutable.format is set to mdmf
sometimes tahoe backup process several files fine and later stops
if it stops - it's always same place
like this
{{{
skipping "/Users/vikarti/Calibre Library/Zooko Wilcox-O'Hearn_ Brian
Warner/Tahoe - The Least-Authority Filesystem (1347)/Tahoe - The Least-
Authority Filesystem - Zooko Wilcox-O'Hearn_ Brian Warner.pdf"..
re-using old directory for "/Users/vikarti/Calibre Library/Zooko
Wilcox-O'Hearn_ Brian Warner/Tahoe - The Least-Authority Filesystem
(1347)"
re-using old directory for "/Users/vikarti/Calibre Library/Zooko
Wilcox-O'Hearn_ Brian Warner"
re-using old directory for '/Users/vikarti/Calibre Library'
Traceback (most recent call last):
File "/Users/vikarti/tahoe-new/allmydata-tahoe-1.9.1/support/bin/tahoe",
line 9, in <module>
load_entry_point('allmydata-tahoe==1.9.1', 'console_scripts',
'tahoe')()
File "/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/scripts/runner.py", line 113, in run
rc = runner(sys.argv[1:], install_node_control=install_node_control)
File "/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/scripts/runner.py", line 99, in runner
rc = cli.dispatch[command](so)
File "/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/scripts/cli.py", line 569, in backup
rc = tahoe_backup.backup(options)
File "/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py", line 325, in backup
return bu.run()
File "/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py", line 123, in run
put_child(archives_url, now, new_backup_dircap)
File "/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py", line 57, in put_child
raise HTTPError("Error during put_child", resp)
allmydata.scripts.common_http.HTTPError: Error during put_child: 500
Internal Server Error
"Traceback (most recent call last):
File
\"/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/base.py\",
line 793, in runUntilCurrent
call.func(*call.args, **call.kw)
File \"/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/support/lib/python2.7/site-
packages/foolscap-0.6.3-py2.7.egg/foolscap/eventual.py\", line 26, in
_turn
cb(*args, **kwargs)\x0a File
\"/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py\",
line 361, in callback
self._startRunCallbacks(result)\x0a File
\"/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py\",
line 455, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File
\"/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/twisted/internet/defer.py\",
line 542, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File \"/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/mutable/filenode.py\", line 855, in <lambda>\x0a
self._modify_once(modifier, first_time))\x0a File \"/Users/vikarti/tahoe-
new/allmydata-tahoe-1.9.1/src/allmydata/mutable/filenode.py\", line 881,
in _modify_once
d = self._try_to_download_data()
File \"/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/mutable/filenode.py\", line 959, in
_try_to_download_data
d = self._read(c, fetch_privkey=True)
File \"/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/mutable/filenode.py\", line 980, in _read
d = r.download(consumer, offset, size)
File \"/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/mutable/retrieve.py\", line 237, in download
self._setup_download()
File \"/Users/vikarti/tahoe-new/allmydata-
tahoe-1.9.1/src/allmydata/mutable/retrieve.py\", line 277, in
_setup_download
shares = versionmap[self.verinfo]\x0aexceptions.KeyError: (5,
'\\x11c\\x1c\\xf7\\xc9\\xae1e\\xa2\\xdaF\\xaf\\xb6S\\x03IH\\x84\\x11#\\xb5\\xf2TS\\xfb\\xde\\x08\\x81\\x01\\xf0\\xd3\\xec',
None, 131073, 1080, 3, 10,
'\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x05\\x11c\\x1c\\xf7\\xc9\\xae1e\\xa2\\xdaF\\xaf\\xb6S\\x03IH\\x84\\x11#\\xb5\\xf2TS\\xfb\\xde\\x08\\x81\\x01\\xf0\\xd3\\xec\\x03\\n\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x048',
(('EOF', 2575), ('verification_key', 1729), ('enc_privkey', 123),
('share_data', 2167), ('signature', 1473), ('block_hash_tree', 2543),
('share_hash_chain', 1337), ('verification_key_end', 2021)))
"
}}}
--
Comment (by daira):
AF saw this bug -- it happened nondeterministically:
{{{
time tahoe cp /tmp/zeros-100m tahoe:zeros3
Error during PUT: 500 Internal Server Error
"Traceback (most recent call last):
File \"/root/allmydata-tahoe-1.10.0/support/lib/python2.7/site-
packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/base.py\",
line 824, in runUntilCurrent
call.func(*call.args, **call.kw)
File \"/root/allmydata-tahoe-1.10.0/support/lib/python2.7/site-
packages/foolscap-0.6.4-py2.7.egg/foolscap/eventual.py\", line 26, in
_turn
cb(*args, **kwargs)
File \"/root/allmydata-tahoe-1.10.0/support/lib/python2.7/site-
packages/Twisted-13.0.0-py2.7-linux-
x86_64.egg/twisted/internet/defer.py\", line 380, in callback
self._startRunCallbacks(result)
File \"/root/allmydata-tahoe-1.10.0/support/lib/python2.7/site-
packages/Twisted-13.0.0-py2.7-linux-
x86_64.egg/twisted/internet/defer.py\", line 488, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File \"/root/allmydata-tahoe-1.10.0/support/lib/python2.7/site-
packages/Twisted-13.0.0-py2.7-linux-
x86_64.egg/twisted/internet/defer.py\", line 575, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File \"/root/allmydata-tahoe-1.10.0/src/allmydata/mutable/filenode.py\",
line 854, in <lambda>
self._modify_once(modifier, first_time))
File \"/root/allmydata-tahoe-1.10.0/src/allmydata/mutable/filenode.py\",
line 880, in _modify_once
d = self._try_to_download_data()
File \"/root/allmydata-tahoe-1.10.0/src/allmydata/mutable/filenode.py\",
line 958, in _try_to_download_data
d = self._read(c, fetch_privkey=True)
File \"/root/allmydata-tahoe-1.10.0/src/allmydata/mutable/filenode.py\",
line 980, in _read
d = r.download(consumer, offset, size)
File \"/root/allmydata-tahoe-1.10.0/src/allmydata/mutable/retrieve.py\",
line 244, in download
self._setup_download()
File \"/root/allmydata-tahoe-1.10.0/src/allmydata/mutable/retrieve.py\",
line 284, in _setup_download
shares = versionmap[self.verinfo]
exceptions.KeyError: (9,
'\\x9b\\xb9\\xaa\\xa5\\xfa\\x88\\x9bX\\x03\\x99HK\\x06\\xed\\xa5V\\x12\\xd6f\\x97l^\\x9f\\x06\\x98\\xac\\xe7\\xb4\\xcdn\\x82\\xa8',
'V\\x97\\x15\\\\\\xf8[7\\rZ\\x9bZ\\xf1\\x93\\xe2\\xdb\\xc2', 2145, 2144,
3, 10,
'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\t\\x9b\\xb9\\xaa\\xa5\\xfa\\x88\\x9bX\\x03\\x99HK\\x06\\xed\\xa5V\\x12\\xd6f\\x97l^\\x9f\\x06\\x98\\xac\\xe7\\xb4\\xcdn\\x82\\xa8V\\x97\\x15\\\\\\xf8[7\\rZ\\x9bZ\\xf1\\x93\\xe2\\xdb\\xc2\\x03\\n\\x00\\x00\\x00\\x00\\x00\\x00\\x08a\\x00\\x00\\x00\\x00\\x00\\x00\\x08\`',
(('enc_privkey', 1538), ('EOF', 2753), ('share_data', 823), ('signature',
399), ('block_hash_tree', 791), ('share_hash_chain', 655)))
"
}}}
(Presumably it is happening on the update of the {{{tahoe:zeros3}}}
directory.)
--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1670#comment:38>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage
More information about the tahoe-lafs-trac-stream
mailing list