[tahoe-dev] Observations on Tahoe performance

[Shawn Willden]

While tuning my GridBackup uploader, I've noticed some things about the 
perfomance of immutable uploads through a helper that puzzle me.  Note that 
I'm only doing immutable uploads, there are no mutable files, not even 
dirnodes.

First, I should say that I haven't yet instrumented the uploader, so these 
are "observations", not "measurements".

The configuration is that GridUpload is using the web API to upload to a local 
Tahoe node which is sending the data through my puny cable modem to a helper 
sitting in a colo on a 2 Gbps connection.  For test purposes I'm using the 
test grid, but I see the same behavior on my friendnet.

The first observation is that when uploading small files it is very hard to 
keep the upstream pipe full, regardless of my configuration.  My upstream 
connection is only about 400 kbps, but I can't sustain any more than about 
200 kbps usage regardless of whether I'm using a local node that goes through 
a helper, direct to the helper, or a local node without the helper.  Of 
course, not using the helper produces the worst effective throughput, because 
the uploaded data volume stays about the same, but the data is FEC-expanded.

Something that seems to help to some degree is to run parallel upload threads.  
All of the threads are talking to the same local Tahoe node, so I'd expect 
that I should be able to maintain roughly the same performance whether the 
uploads are done in parallel or serially -- especially since that node is 
just pumping the data up to the helper.  There is a diminishing returns 
effect on the number of threads, but more seems to be better, up to the point 
where it's too much and starts to trigger 500 errors in the Tahoe node, 
or "too many open files" exceptions.

Any idea why that would be?

The second observation is that performance seems to be somewhat improved by 
sending to a local node which sends to a helper, rather than connecting 
directly to the helper.  There are obviously security reasons to prefer a 
local node, but it seems to help performance as well?

Any ideas about that one?

I'm trying to understand what influences performance and why, so I can 
maximize the upload throughput.  My complete backup will take months to 
upload either way, but two months is better than four :-)

	Shawn.