tahoe backup error

[ianchov]

Hi guys

i do my backup with

/home/ianchov/allmydata-tahoe-1.9.1/bin/tahoe backup -v /home/ianchov/testmount XYZ:XYZ

 ..............
 re-using old directory for '/home/ianchov/testmount/dir22'
 creating directory for '/home/ianchov/testmount'
Traceback (most recent call last):
  File "/home/ianchov/allmydata-tahoe-1.9.1/support/bin/tahoe", line 9, in <module>
    load_entry_point('allmydata-tahoe==1.9.1', 'console_scripts', 'tahoe')()
  File "/home/ianchov/allmydata-tahoe-1.9.1/src/allmydata/scripts/runner.py", line 113, in run
    rc = runner(sys.argv[1:], install_node_control=install_node_control)
  File "/home/ianchov/allmydata-tahoe-1.9.1/src/allmydata/scripts/runner.py", line 99, in runner
    rc = cli.dispatch[command](so)
  File "/home/ianchov/allmydata-tahoe-1.9.1/src/allmydata/scripts/cli.py", line 569, in backup
    rc = tahoe_backup.backup(options)
  File "/home/ianchov/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py", line 325, in backup
    return bu.run()
  File "/home/ianchov/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py", line 123, in run
    put_child(archives_url, now, new_backup_dircap)
  File "/home/ianchov/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py", line 57, in put_child
    raise HTTPError("Error during put_child", resp)
allmydata.scripts.common_http.HTTPError: Error during put_child: 500 Internal Server Error
"Traceback (most recent call last):\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/support/lib/python2.7/site-packages/Twisted-10.1.0-py2.7-linux-x86_64.egg/twisted/internet/base.py\", line 796, in runUntilCurrent\x0a    call.func(*call.args, **call.kw)\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/support/lib/python2.7/site-packages/foolscap-0.6.2-py2.7.egg/foolscap/eventual.py\", line 26, in _turn\x0a    cb(*args, **kwargs)\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/support/lib/python2.7/site-packages/Twisted-10.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py\", line 318, in callback\x0a    self._startRunCallbacks(result)\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/support/lib/python2.7/site-packages/Twisted-10.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py\", line 424, in _startRunCallbacks\x0a    self._runCallbacks()\x0a--- <exception caught here> ---\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/support/lib/python2.7/site-packages/Twisted-10.1.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py\", line 441, in _runCallbacks\x0a    self.result = callback(self.result, *args, **kw)\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/src/allmydata/mutable/filenode.py\", line 855, in <lambda>\x0a    self._modify_once(modifier, first_time))\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/src/allmydata/mutable/filenode.py\", line 881, in _modify_once\x0a    d = self._try_to_download_data()\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/src/allmydata/mutable/filenode.py\", line 959, in _try_to_download_data\x0a    d = self._read(c, fetch_privkey=True)\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/src/allmydata/mutable/filenode.py\", line 980, in _read\x0a    d = r.download(consumer, offset, size)\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/src/allmydata/mutable/retrieve.py\", line 237, in download\x0a    self._setup_download()\x0a  File \"/home/ianchov/allmydata-tahoe-1.9.0/src/allmydata/mutable/retrieve.py\", line 277, in _setup_download\x0a    shares = versionmap[self.verinfo]\x0aexceptions.KeyError: (44, '\\x94\\x17\\xb3]\\xb7%\\r\\xbe\\xb4\\xe6\\xa1\\x8d\\xbf\\x04\\x85\\x8f\\x9f\\xd4\\xb3\\x94\\xbf\\xab\\x15N\\x9c\\xb8E\\xd9\\xc7\\xdb:\\xef', '\\xcf\\x10\\xf6\$HO\\xdc\\xc3i\\xbc\\xeb\\xb7\\xed\\xfd\\xa2\\xac', 8240, 8238, 5, 12, '\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00,\\x94\\x17\\xb3]\\xb7%\\r\\xbe\\xb4\\xe6\\xa1\\x8d\\xbf\\x04\\x85\\x8f\\x9f\\xd4\\xb3\\x94\\xbf\\xab\\x15N\\x9c\\xb8E\\xd9\\xc7\\xdb:\\xef\\xcf\\x10\\xf6\$HO\\xdc\\xc3i\\xbc\\xeb\\xb7\\xed\\xfd\\xa2\\xac\\x05\\x0c\\x00\\x00\\x00\\x00\\x00\\x00 0\\x00\\x00\\x00\\x00\\x00\\x00 .', (('enc_privkey', 2471), ('EOF', 3688), ('share_data', 823), ('signature', 399), ('block_hash_tree', 791), ('share_hash_chain', 655)))\x0a"

[gyver]

I have the very same exception (KeyError), happening at the same spot ( shares = versionmap[self.verinfo] ). I use tahoe cp for my backups though. I can replicate this as soon as I try to backup a file in one of two directories which are direct subdirs of an alias I can't "tahoe deep-check --repair" anymore (I'll open a separate bug). The file being cp'ed is correctly created and can be retrieved intact even though tahoe receives a 500 error.

On the same private tahoe storage network another alias doesn't have this problem : I can tahoe cp and repair without errors so I guess the problem is either at the alias or both of its 2 subdirs. Note that I tahoe cp huge files and sometimes one flaky storage node ADSL provider can break the connection in middle of a transfer.

[gyver]

Replying to gyver: couldn't find out how to attach an incident file, uploading one returns me to a page asking me if I want to add other attachments...

[davidsarah]

Duplicate of #1670 (since the KeyError occurs on the same line).

gyver: that problem can happen if the attachment is too large. How large is the file? Can you host it somewhere else?

(You can reply here even though this bug is closed as a duplicate.)

[gyver]

The problem isn't due to the file size : I manage to backup files up to 3GB on the same storage network from the same point but in other directories and the failures happen with 8MB files.

[gyver]

Replying to gyver:

The problem isn't due to the file size : I manage to backup files up to 3GB on the same storage network from the same point but in other directories and the failures happen with 8MB files.

Oops, I think I just understood which attachment you were referring to. The incident file was around 40kB IIRC. I'll host it somewhere.

[gyver]

Here is an incident file for the tahoe put

​http://www.bouton.name/incident-2012-02-18--22_00_17Z-7jusaxa.flog.bz2

Here is an incident file for the tahoe deep-check --repair in the same problematic "tree"

​http://www.bouton.name/incident-2012-02-18--22_59_20Z-fsyvi7q.flog.bz2

[davidsarah]

​http://www.bouton.name/incident-2012-02-18--22_00_17Z-7jusaxa.flog.bz2 gives me a 403 Forbidden (probably incorrect file permissions).

[gyver]

Sorry, I plead exhaustion: apparently I did chmod a+x instead of a+r for this one... Fixed.

[davidsarah]

Okay, I've attached those files to #1670. Thanks.

[ianchov]

Ok The error dissappeard as long i just change the shares.total = 15 before it was to 12...

????