[volunteergrid2-l] upload speeds

[Shawn Willden]

On Thu, Apr 12, 2012 at 9:10 PM, erpo41 at gmail.com <erpo41 at gmail.com> wrote:

> On Thu, Apr 12, 2012 at 9:00 PM, Shawn Willden <shawn at willden.org> wrote:
>
>> On Thu, Apr 12, 2012 at 8:09 PM, erpo41 at gmail.com <erpo41 at gmail.com>wrote:
>>
>>>
>>>>> *I don't think it's my upstream connection since I have about 1000Kbps
>>> upstream capacity and I'm not using it for anything else.
>>> *I don't think it's everyone else's downstream connection because those
>>> are typically way faster than 200Kbps.
>>>
>>
>> Do you mean Kbps or KBps (bits or bytes)?  If your upstream is 1000 Kbps
>> or 1 Mbps, then your max upload speed would be about 100 KBps, so I'm
>> assuming you must mean KBps.
>>
>
> I mean 200,000 bits per second.
>
>
>> Terminology quibble out of the way, the Tahoe uploader is somewhat
>> stupid, as I understand it, and effectively limits your upload speed to
>> that of the slowest node, and then limits it some more due to inter-message
>> waiting delays.
>>
>
> That might explain it. Based on the error messages I've seen so far, I
> suspect that the tahoe CLI goes through the WUI doing one PUT request at a
> time. That PUT request shouldn't return until the file is totally uploaded
> including all erasure coding data to all servers, so it would make sense
> that the slowest node would place a lower limit on the transfer time.
>

Yes, but it's a little worse than that.  Tahoe breaks the file into chunks,
then breaks each chunk into shares, then pushes the shares in a
round-robin-ish fashion (I believe -- it's been a long time since I looked
at the code and I'm not sure I ever understood it fully).  So it uploads
more or less in lockstep, with lots of inter-message delays to slow things
down.

It was originally implemented in the context of a data center with gigabit
inter-node bandwidth and sub-millisecond inter-node latencies.  A helper
sitting in that same environment was pushing shares to the storage nodes,
and the data was coming to the helper over the Internet -- typically
bottlenecked on an itty bitty ADSL uplink.  So optimizing the upload
process wasn't a priority.  The helper could easily take the data as fast
as clients could push it.

That's a very different environment than our servers scattered all over the
globe.  Much faster, but much less fault-tolerant.


>
>
>> I'm currently sustaining about 3.3 Mbps upload rate (measured at the NIC)
>> which translates to net upload speeds (reported by Tahoe) of around 230
>> KBps.  Since I'm using N=19, K=11, my expansion factor is 1.72, which turns
>> 230 KBps into 396 KBps, which is 3164 Kbps.
>>
>
> My 200Kbps is measured at the NIC also. How do I pull stats out of tahoe
> directly?
>

Go to your web interface.  In the upper-right corner is a link called
"Recent Uploads and Downloads".  On that page you'll see your recent file
transfers and you can click on one of them to look at stats.


>
>
>> That's far below my maximum upstream speed (~50 Mbps),
>>
>
> Braggart. :)
>

It is nice!  I'm on a Comcast Business Class cable modem connection.


>
>
>> but if I can sustain 200 KBps, I can upload 17 GB per day, which is
>> perfectly adequate for my needs.
>>
>
> Hey, 200Kbps meets my needs just fine; it's more bandwidth to offsite
> backup than I had before I joined the grid. But if I can get more I'd like
> it.
>

More is clearly better.


>
>
>> My bigger problem is that with N=19 and only 20 nodes accepting shares
>> even when everything is up, my uploads tend to get stopped a lot.
>>
>>
> This is why I picked N=15, H=10, K=5. So far, no issues in that area.
>

I just use:

while true; do
   tahoe backup ...
   sleep 60
done

So when the servers are up, my backups run.  H=N=19 is a little bit
optimistic, perhaps, but I'm expecting to add some more nodes soon, and one
thing you can't easily do is change the encoding of already-uploaded files.
 When you repair them, they repair to the parameters specified during their
original upload, no better.

-- 
Shawn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20120412/a930d988/attachment.html>