[volunteergrid2-l] adding files to the grid

[Billy Earney]

Zooko,

On regular file uploads, how can I tell which server is causing the
bottleneck?  On some special uploads such as map updates, it breaks stats
down by server, but for regular files, I get data like so:

File Upload Status

   - Started: 17:20:25 02-Mar-2011
   - Storage Index: 5nqo5puy44bvkrvyrwbgobchbq
   - Helper?: No
   - Total Size: 4945920
   - Progress (Hash): 100.0%
   - Progress (Ciphertext): 100.0%
   - Progress (Encode+Push): 100.0%
   - Status: Finished

 Upload Results

   - Shares Pushed: 10
   - Shares Already Present: 0
   - Sharemap:
      - 0 -> placed on [gbytbnxw]
      - 1 -> placed on [qivwuhf6]
      - 2 -> placed on [ezcaxvy7]
      - 3 -> placed on [jbrse33y]
      - 4 -> placed on [g4xvpwqa]
      - 5 -> placed on [voej6qtg]
      - 6 -> placed on [pmitrhwg]
      - 7 -> placed on [vifimgiw]
      - 8 -> placed on [gbytbnxw]
      - 9 -> placed on [qivwuhf6]
   - Servermap:
      - [ezcaxvy7] got share: #2
      - [gbytbnxw] got shares: #0,#8
      - [g4xvpwqa] got share: #4
      - [jbrse33y] got share: #3
      - [pmitrhwg] got share: #6
      - [qivwuhf6] got shares: #1,#9
      - [vifimgiw] got share: #7
      - [voej6qtg] got share: #5
   - Timings:
      - File Size: 4945920 bytes
      - Total: 3 minutes (21.9kBps)
         - Storage Index: 41ms (119.72MBps)
         - [Contacting Helper]:
            - [Helper Already-In-Grid Check]:
         - [Upload Ciphertext To Helper]: ()
         - Peer Selection: 1.31s
         - Encode And Push: 3 minutes (22.3kBps)
            - Cumulative Encoding: 475ms (10.41MBps)
            - Cumulative Pushing: 3 minutes (22.3kBps)
            - Send Hashes And Close: 2.51s
         - [Helper Total]:


On Wed, Mar 2, 2011 at 4:39 PM, Billy Earney <billy.earney at gmail.com> wrote:

> Steve,
>
> 9am this morning is the last time I saw your server..  My process is
> currently loading larger files (a few hundred MB per file) and at 50kbps,
> its taking a few hours per file.   I'm guessing in the next hour, another
> file will be uploaded and I can see if your server participates in the
> upload.
>
> Below is some stats for the last time my machine uploaded shares to your
> machine.
>
>
>    - Timings:
>       - Total: 2 minutes
>          - Initial Queries: 7.8ms
>          - Cumulative Verify: 3.3ms
>       - Per-Server Response Times:
>          - [gbytbnxw]: 2.41s
>          - [g4xvpwqa]: 2 minutes
>          - [jbrse33y]: 2.03s
>          - [pmitrhwg]: 2.27s
>          - [qivwuhf6]: 2.33s
>          - [vifimgiw]: 25ms
>          - [voej6qtg]: 2.30s
>
>
>
>
> On Wed, Mar 2, 2011 at 8:27 AM, Zooko O'Whielacronx <zooko at zooko.com>wrote:
>
>> On Wed, Mar 2, 2011 at 6:57 AM, Billy Earney <billy.earney at gmail.com>
>> wrote:
>> >
>> > I want to thank you for the help last night on IRC,  I had to go, so I
>> > couldn't wait for you to return..  Looking at the stats from the recent
>> > upload and download operations, it looks like the encoding is doing
>> about
>> > 10MBps (which is very good), but the uploading portion is only doing
>> around
>> > 30Kbps, so it looks like the upload is the bottleneck.  My upload speed
>> is
>> > near (1Mbps), I checked this on speedtest.net, and it seems to be
>> accurate.
>> > I don't expect to get 1Mbps upload speed with tahoe-lafs since it has
>> some
>> > extra overhead, but I would expect at least 200KBps or so.
>>
>> Cool! I think those stats should show you details about which servers
>> you were uploading to and how each one performed. I suspect that there
>> is one server which is accepting data very slowly.
>>
>> The uploader uploads to the servers in parallel, so it proceeds at
>> exactly the speed of the slowest server.
>>
>> Regards,
>>
>> Zooko
>> _______________________________________________
>> volunteergrid2-l mailing list
>> volunteergrid2-l at tahoe-lafs.org
>> http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
>> http://bigpig.org/twiki/bin/view/Main/WebHome
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20110302/a4bf6395/attachment.html>