[volunteergrid2-l] speeds again, and missing files
Shawn Willden
shawn at willden.org
Fri Apr 20 02:16:31 UTC 2012
Can you post this on tahoe-dev?
On Thu, Apr 19, 2012 at 5:13 PM, erpo41 at gmail.com <erpo41 at gmail.com> wrote:
> It did it again. Here's the trace.
>
> Traceback (most recent call last):
> File "/home/eric/allmydata-tahoe-1.9.1/support/bin/tahoe", line 9, in
> <module>
> load_entry_point('allmydata-tahoe==1.9.1', 'console_scripts',
> 'tahoe')()
> File
> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/runner.py",
> line 113, in run
> rc = runner(sys.argv[1:], install_node_control=install_node_control)
> File
> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/runner.py",
> line 99, in runner
> rc = cli.dispatch[command](so)
> File
> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/cli.py",
> line 569, in backup
> rc = tahoe_backup.backup(options)
> File
> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
> line 325, in backup
> return bu.run()
> File
> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
> line 118, in run
> new_backup_dircap = self.process(options.from_dir)
> File
> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
> line 194, in process
> childcap, metadata = self.upload(childpath)
> File
> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
> line 305, in upload
> raise HTTPError("Error during file PUT", resp)
> allmydata.scripts.common_http.HTTPError: Error during file PUT: 500
> Internal Server Error
> "Traceback (most recent call last):\x0a File
> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/immutable/encode.py\",
> line 437, in _send_segment\x0a d = self.send_block(shareid, segnum,
> block, lognum)\x0a File
> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/immutable/encode.py\",
> line 474, in send_block\x0a \"segnum=%d\" % segment_num)\x0a File
> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
> line 312, in addErrback\x0a errbackKeywords=kw)\x0a File
> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
> line 290, in addCallbacks\x0a self._runCallbacks()\x0a--- <exception
> caught here> ---\x0a File
> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
> line 551, in _runCallbacks\x0a current.result = callback(current.result,
> *args, **kw)\x0a File
> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/immutable/encode.py\",
> line 501, in _remove_shareholder\x0a raise
> UploadUnhappinessError(msg)\x0aallmydata.interfaces.UploadUnhappinessError:
> shares could be placed on only 12 server(s) such that any 5 of them have
> enough shares to recover the file, but we were asked to place shares on at
> least 13 such servers.: [Failure instance: Traceback (failure with no
> frames): <class 'allmydata.util.pipeline.PipelineError'>: <PipelineError
> error=([Failure instance: Traceback: <class
> 'foolscap.ipb.DeadReferenceError'>: Calling Stale
> Broker\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/immutable/layout.py:237:_write\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/util/pipeline.py:89:add\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py:134:maybeDeferred\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/foolscap-0.6.3-py2.6.egg/foolscap/referenceable.py:415:callRemote\x0a---
> <exception caught here>
> ---\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py:134:maybeDeferred\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/foolscap-0.6.3-py2.6.egg/foolscap/referenceable.py:455:_callRemote\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/foolscap-0.6.3-py2.6.egg/foolscap/broker.py:477:newRequestID\x0a])>\x0a]\x0a"
>
>
> On Wed, Apr 18, 2012 at 9:43 PM, Shawn Willden <shawn at willden.org> wrote:
>
>> Hmm. Out of 19 servers which are accepting shares, your node could only
>> place shares on 12. This message is odd, though. Usually it also tells
>> how many servers it queried.
>>
>> Maybe Brian or David-Sarah will be willing to look into this?
>>
>>
>> On Wed, Apr 18, 2012 at 5:28 PM, erpo41 at gmail.com <erpo41 at gmail.com>wrote:
>>
>>> Here's a stack trace from the most recent error. Although most uploads
>>> proceed without errors (just very slowly).
>>>
>>> eric at io:/srv/per-user/eric$ tahoe backup backup-T61p/ tahoe:
>>> Traceback (most recent call last):
>>> File "/home/eric/allmydata-tahoe-1.9.1/support/bin/tahoe", line 9, in
>>> <module>
>>> load_entry_point('allmydata-tahoe==1.9.1', 'console_scripts',
>>> 'tahoe')()
>>> File
>>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/runner.py",
>>> line 113, in run
>>> rc = runner(sys.argv[1:], install_node_control=install_node_control)
>>> File
>>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/runner.py",
>>> line 99, in runner
>>> rc = cli.dispatch[command](so)
>>> File
>>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/cli.py",
>>> line 569, in backup
>>> rc = tahoe_backup.backup(options)
>>> File
>>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
>>> line 325, in backup
>>> return bu.run()
>>> File
>>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
>>> line 118, in run
>>> new_backup_dircap = self.process(options.from_dir)
>>> File
>>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
>>> line 194, in process
>>> childcap, metadata = self.upload(childpath)
>>> File
>>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
>>> line 305, in upload
>>> raise HTTPError("Error during file PUT", resp)
>>> allmydata.scripts.common_http.HTTPError: Error during file PUT: 500
>>> Internal Server Error
>>> "Traceback (most recent call last):\x0a File
>>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
>>> line 551, in _runCallbacks\x0a current.result = callback(current.result,
>>> *args, **kw)\x0a File
>>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/util/pipeline.py\",
>>> line 121, in _call_finished\x0a d.errback(self.failure)\x0a File
>>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
>>> line 397, in errback\x0a self._startRunCallbacks(fail)\x0a File
>>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
>>> line 464, in _startRunCallbacks\x0a self._runCallbacks()\x0a---
>>> <exception caught here> ---\x0a File
>>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
>>> line 551, in _runCallbacks\x0a current.result = callback(current.result,
>>> *args, **kw)\x0a File
>>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/immutable/encode.py\",
>>> line 501, in _remove_shareholder\x0a raise
>>> UploadUnhappinessError(msg)\x0aallmydata.interfaces.UploadUnhappinessError:
>>> shares could be placed on only 12 server(s) such that any 5 of them have
>>> enough shares to recover the file, but we were asked to place shares on at
>>> least 13 such servers.: [Failure instance: Traceback (failure with no
>>> frames): <class 'allmydata.util.pipeline.PipelineError'>: <PipelineError
>>> error=([Failure instance: Traceback (failure with no frames): <class
>>> 'foolscap.ipb.DeadReferenceError'>: Connection was lost (to tubid=oted)
>>> (during method=RIBucketWriter:write)\x0a])>\x0a]\x0a"
>>>
>>>
>>> On Wed, Apr 18, 2012 at 12:26 PM, Shawn Willden <shawn at willden.org>wrote:
>>>
>>>> A good start would be to get stack traces from when the errors occur.
>>>>
>>>>
>>>> On Wed, Apr 18, 2012 at 12:20 PM, erpo41 at gmail.com <erpo41 at gmail.com>wrote:
>>>>
>>>>> How do we do this?
>>>>>
>>>>>
>>>>> On Tue, Apr 17, 2012 at 9:08 PM, Shawn Willden <shawn at willden.org>wrote:
>>>>>
>>>>>> I haven't seen these problems. We should look into them more deeply
>>>>>> and figure out what's going on.
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 17, 2012 at 7:13 PM, Billy Earney <billy.earney at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I've had issues too with the grid, many uncorridnatioed writes, and
>>>>>>> errors while deep checking, which terminates the leasing process. I've
>>>>>>> lost some files on the grid (luckily, I've had copies elsewhere). Do
>>>>>>> others have the same experience, and if not, what are some good practices I
>>>>>>> should start using?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> On Tue, Apr 17, 2012 at 7:58 PM, erpo41 at gmail.com <erpo41 at gmail.com>wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I just checked my recent upload and download activity, and I found
>>>>>>>> the following:
>>>>>>>>
>>>>>>>> - Timings:
>>>>>>>> - File Size: 26257857 bytes
>>>>>>>> - Total: 4 hours (1.6kBps)
>>>>>>>> - Storage Index: 293ms (89.56MBps)
>>>>>>>> - [Contacting Helper]:
>>>>>>>> - [Helper Already-In-Grid Check]:
>>>>>>>> - [Upload Ciphertext To Helper]: ()
>>>>>>>> - Peer Selection: 12 seconds
>>>>>>>> - Encode And Push: 4 hours (1.6kBps)
>>>>>>>> - Cumulative Encoding: 2.53s (10.37MBps)
>>>>>>>> - Cumulative Pushing: 4 hours (1.6kBps)
>>>>>>>> - Send Hashes And Close: 33 seconds
>>>>>>>> - [Helper Total]:
>>>>>>>>
>>>>>>>> Does 1.6kBps sound about right?
>>>>>>>>
>>>>>>>> Also, I've been using the command "tahoe backup backup-T61p
>>>>>>>> tahoe:". When I use the WUI to navigate to the writecap associated with
>>>>>>>> tahoe:, it shows the only entry in the directory is another directory
>>>>>>>> called Archive. The Archive directory is empty, even though I've been
>>>>>>>> uploading files (according to upload activity log). What gives?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> volunteergrid2-l mailing list
>>>>>>>> volunteergrid2-l at tahoe-lafs.org
>>>>>>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>>>>>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> volunteergrid2-l mailing list
>>>>>>> volunteergrid2-l at tahoe-lafs.org
>>>>>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>>>>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Shawn
>>>>>>
>>>>>> _______________________________________________
>>>>>> volunteergrid2-l mailing list
>>>>>> volunteergrid2-l at tahoe-lafs.org
>>>>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>>>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> volunteergrid2-l mailing list
>>>>> volunteergrid2-l at tahoe-lafs.org
>>>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Shawn
>>>>
>>>> _______________________________________________
>>>> volunteergrid2-l mailing list
>>>> volunteergrid2-l at tahoe-lafs.org
>>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>>
>>>
>>>
>>> _______________________________________________
>>> volunteergrid2-l mailing list
>>> volunteergrid2-l at tahoe-lafs.org
>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>
>>
>>
>>
>> --
>> Shawn
>>
>> _______________________________________________
>> volunteergrid2-l mailing list
>> volunteergrid2-l at tahoe-lafs.org
>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>
>
>
> _______________________________________________
> volunteergrid2-l mailing list
> volunteergrid2-l at tahoe-lafs.org
> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
> http://bigpig.org/twiki/bin/view/Main/WebHome
>
--
Shawn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20120419/63dd45a0/attachment-0001.html>
More information about the volunteergrid2-l
mailing list