[volunteergrid2-l] speeds again, and missing files
erpo41 at gmail.com
erpo41 at gmail.com
Thu Apr 19 23:13:47 UTC 2012
It did it again. Here's the trace.
Traceback (most recent call last):
File "/home/eric/allmydata-tahoe-1.9.1/support/bin/tahoe", line 9, in
<module>
load_entry_point('allmydata-tahoe==1.9.1', 'console_scripts', 'tahoe')()
File
"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/runner.py",
line 113, in run
rc = runner(sys.argv[1:], install_node_control=install_node_control)
File
"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/runner.py",
line 99, in runner
rc = cli.dispatch[command](so)
File
"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/cli.py",
line 569, in backup
rc = tahoe_backup.backup(options)
File
"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
line 325, in backup
return bu.run()
File
"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
line 118, in run
new_backup_dircap = self.process(options.from_dir)
File
"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
line 194, in process
childcap, metadata = self.upload(childpath)
File
"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
line 305, in upload
raise HTTPError("Error during file PUT", resp)
allmydata.scripts.common_http.HTTPError: Error during file PUT: 500
Internal Server Error
"Traceback (most recent call last):\x0a File
\"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/immutable/encode.py\",
line 437, in _send_segment\x0a d = self.send_block(shareid, segnum,
block, lognum)\x0a File
\"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/immutable/encode.py\",
line 474, in send_block\x0a \"segnum=%d\" % segment_num)\x0a File
\"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
line 312, in addErrback\x0a errbackKeywords=kw)\x0a File
\"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
line 290, in addCallbacks\x0a self._runCallbacks()\x0a--- <exception
caught here> ---\x0a File
\"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
line 551, in _runCallbacks\x0a current.result = callback(current.result,
*args, **kw)\x0a File
\"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/immutable/encode.py\",
line 501, in _remove_shareholder\x0a raise
UploadUnhappinessError(msg)\x0aallmydata.interfaces.UploadUnhappinessError:
shares could be placed on only 12 server(s) such that any 5 of them have
enough shares to recover the file, but we were asked to place shares on at
least 13 such servers.: [Failure instance: Traceback (failure with no
frames): <class 'allmydata.util.pipeline.PipelineError'>: <PipelineError
error=([Failure instance: Traceback: <class
'foolscap.ipb.DeadReferenceError'>: Calling Stale
Broker\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/immutable/layout.py:237:_write\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/util/pipeline.py:89:add\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py:134:maybeDeferred\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/foolscap-0.6.3-py2.6.egg/foolscap/referenceable.py:415:callRemote\x0a---
<exception caught here>
---\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py:134:maybeDeferred\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/foolscap-0.6.3-py2.6.egg/foolscap/referenceable.py:455:_callRemote\x0a/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/foolscap-0.6.3-py2.6.egg/foolscap/broker.py:477:newRequestID\x0a])>\x0a]\x0a"
On Wed, Apr 18, 2012 at 9:43 PM, Shawn Willden <shawn at willden.org> wrote:
> Hmm. Out of 19 servers which are accepting shares, your node could only
> place shares on 12. This message is odd, though. Usually it also tells
> how many servers it queried.
>
> Maybe Brian or David-Sarah will be willing to look into this?
>
>
> On Wed, Apr 18, 2012 at 5:28 PM, erpo41 at gmail.com <erpo41 at gmail.com>wrote:
>
>> Here's a stack trace from the most recent error. Although most uploads
>> proceed without errors (just very slowly).
>>
>> eric at io:/srv/per-user/eric$ tahoe backup backup-T61p/ tahoe:
>> Traceback (most recent call last):
>> File "/home/eric/allmydata-tahoe-1.9.1/support/bin/tahoe", line 9, in
>> <module>
>> load_entry_point('allmydata-tahoe==1.9.1', 'console_scripts',
>> 'tahoe')()
>> File
>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/runner.py",
>> line 113, in run
>> rc = runner(sys.argv[1:], install_node_control=install_node_control)
>> File
>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/runner.py",
>> line 99, in runner
>> rc = cli.dispatch[command](so)
>> File
>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/cli.py",
>> line 569, in backup
>> rc = tahoe_backup.backup(options)
>> File
>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
>> line 325, in backup
>> return bu.run()
>> File
>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
>> line 118, in run
>> new_backup_dircap = self.process(options.from_dir)
>> File
>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
>> line 194, in process
>> childcap, metadata = self.upload(childpath)
>> File
>> "/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/scripts/tahoe_backup.py",
>> line 305, in upload
>> raise HTTPError("Error during file PUT", resp)
>> allmydata.scripts.common_http.HTTPError: Error during file PUT: 500
>> Internal Server Error
>> "Traceback (most recent call last):\x0a File
>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
>> line 551, in _runCallbacks\x0a current.result = callback(current.result,
>> *args, **kw)\x0a File
>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/util/pipeline.py\",
>> line 121, in _call_finished\x0a d.errback(self.failure)\x0a File
>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
>> line 397, in errback\x0a self._startRunCallbacks(fail)\x0a File
>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
>> line 464, in _startRunCallbacks\x0a self._runCallbacks()\x0a---
>> <exception caught here> ---\x0a File
>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-i686.egg/twisted/internet/defer.py\",
>> line 551, in _runCallbacks\x0a current.result = callback(current.result,
>> *args, **kw)\x0a File
>> \"/home/volunteergrid2/allmydata-tahoe-1.9.1/src/allmydata/immutable/encode.py\",
>> line 501, in _remove_shareholder\x0a raise
>> UploadUnhappinessError(msg)\x0aallmydata.interfaces.UploadUnhappinessError:
>> shares could be placed on only 12 server(s) such that any 5 of them have
>> enough shares to recover the file, but we were asked to place shares on at
>> least 13 such servers.: [Failure instance: Traceback (failure with no
>> frames): <class 'allmydata.util.pipeline.PipelineError'>: <PipelineError
>> error=([Failure instance: Traceback (failure with no frames): <class
>> 'foolscap.ipb.DeadReferenceError'>: Connection was lost (to tubid=oted)
>> (during method=RIBucketWriter:write)\x0a])>\x0a]\x0a"
>>
>>
>> On Wed, Apr 18, 2012 at 12:26 PM, Shawn Willden <shawn at willden.org>wrote:
>>
>>> A good start would be to get stack traces from when the errors occur.
>>>
>>>
>>> On Wed, Apr 18, 2012 at 12:20 PM, erpo41 at gmail.com <erpo41 at gmail.com>wrote:
>>>
>>>> How do we do this?
>>>>
>>>>
>>>> On Tue, Apr 17, 2012 at 9:08 PM, Shawn Willden <shawn at willden.org>wrote:
>>>>
>>>>> I haven't seen these problems. We should look into them more deeply
>>>>> and figure out what's going on.
>>>>>
>>>>>
>>>>> On Tue, Apr 17, 2012 at 7:13 PM, Billy Earney <billy.earney at gmail.com>wrote:
>>>>>
>>>>>> I've had issues too with the grid, many uncorridnatioed writes, and
>>>>>> errors while deep checking, which terminates the leasing process. I've
>>>>>> lost some files on the grid (luckily, I've had copies elsewhere). Do
>>>>>> others have the same experience, and if not, what are some good practices I
>>>>>> should start using?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> On Tue, Apr 17, 2012 at 7:58 PM, erpo41 at gmail.com <erpo41 at gmail.com>wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I just checked my recent upload and download activity, and I found
>>>>>>> the following:
>>>>>>>
>>>>>>> - Timings:
>>>>>>> - File Size: 26257857 bytes
>>>>>>> - Total: 4 hours (1.6kBps)
>>>>>>> - Storage Index: 293ms (89.56MBps)
>>>>>>> - [Contacting Helper]:
>>>>>>> - [Helper Already-In-Grid Check]:
>>>>>>> - [Upload Ciphertext To Helper]: ()
>>>>>>> - Peer Selection: 12 seconds
>>>>>>> - Encode And Push: 4 hours (1.6kBps)
>>>>>>> - Cumulative Encoding: 2.53s (10.37MBps)
>>>>>>> - Cumulative Pushing: 4 hours (1.6kBps)
>>>>>>> - Send Hashes And Close: 33 seconds
>>>>>>> - [Helper Total]:
>>>>>>>
>>>>>>> Does 1.6kBps sound about right?
>>>>>>>
>>>>>>> Also, I've been using the command "tahoe backup backup-T61p
>>>>>>> tahoe:". When I use the WUI to navigate to the writecap associated with
>>>>>>> tahoe:, it shows the only entry in the directory is another directory
>>>>>>> called Archive. The Archive directory is empty, even though I've been
>>>>>>> uploading files (according to upload activity log). What gives?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> volunteergrid2-l mailing list
>>>>>>> volunteergrid2-l at tahoe-lafs.org
>>>>>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>>>>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> volunteergrid2-l mailing list
>>>>>> volunteergrid2-l at tahoe-lafs.org
>>>>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>>>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Shawn
>>>>>
>>>>> _______________________________________________
>>>>> volunteergrid2-l mailing list
>>>>> volunteergrid2-l at tahoe-lafs.org
>>>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> volunteergrid2-l mailing list
>>>> volunteergrid2-l at tahoe-lafs.org
>>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>>
>>>
>>>
>>>
>>> --
>>> Shawn
>>>
>>> _______________________________________________
>>> volunteergrid2-l mailing list
>>> volunteergrid2-l at tahoe-lafs.org
>>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>>
>>
>>
>> _______________________________________________
>> volunteergrid2-l mailing list
>> volunteergrid2-l at tahoe-lafs.org
>> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>
>
>
>
> --
> Shawn
>
> _______________________________________________
> volunteergrid2-l mailing list
> volunteergrid2-l at tahoe-lafs.org
> https://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
> http://bigpig.org/twiki/bin/view/Main/WebHome
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20120419/a18b3aee/attachment-0001.html>
More information about the volunteergrid2-l
mailing list