Hello,<br><br>I was very happy to see this elephant in the room finally discussed.<br><br>> Its inability to
saturate a good upstream pipe is probably the biggest reason that I<br>> sometimes hesitate to recommend it as a serious backup solution to my
friends.<br><br>This is exactly what I was forced to admit, except that in my case recommendations go to my consulting clients. I am still here, as I see enormous potential in Tahoe, but for it to be realized, this project will need more users in the real-world scenarios.<br>
<br>And for that to happen, this performance issue should be the absolutely of highest priority. <br><br>Thanks,<br clear="all">Andrej Falout<br><br>
<br><br><div class="gmail_quote">On Thu, Aug 20, 2009 at 6:20 AM, Brian Warner <span dir="ltr"><<a href="mailto:warner@lothar.com">warner@lothar.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">Shawn Willden wrote:<br>
> On Wednesday 12 August 2009 08:02:13 am Francois Deppierraz wrote:<br>
>> Hi Shawn,<br>
>><br>
>> Shawn Willden wrote:<br>
>>> The first observation is that when uploading small files it is very<br>
>>> hard to keep the upstream pipe full, regardless of my configuration.<br>
>>> My upstream connection is only about 400 kbps, but I can't sustain<br>
>>> any more than about 200 kbps usage regardless of whether I'm using a<br>
>>> local node that goes through a helper, direct to the helper, or a<br>
>>> local node without the helper. Of course, not using the helper<br>
>>> produces the worst effective throughput, because the uploaded data<br>
>>> volume stays about the same, but the data is FEC-expanded.<br>
>> This is probably because Tahoe doesn't use a windowing protocol which<br>
>> makes it especially sensitive to the latency between nodes.<br>
>><br>
>> You should have a look at Brian's description in bug #397.<br>
><br>
> Very interesting, but I don't think #397 explains what I'm seeing. I<br>
> can pretty well saturate my upstream connection with Tahoe when<br>
> uploading a large file.<br>
<br>
</div>First off, I must admit that I'm somewhat embarrassed by Tahoe's<br>
relatively poor throughput+latency performance. Its inability to<br>
saturate a good upstream pipe is probably the biggest reason that I<br>
sometimes hesitate to recommend it as a serious backup solution to my<br>
friends. In the <a href="http://allmydata.com" target="_blank">allmydata.com</a> world, we deferred performance analysis<br>
and improvement for two reasons: most consumer upload speeds were really<br>
bad (so we could hide behind that), and we don't have any<br>
bandwidth-management tools to avoid saturating their upstream and thus<br>
causing problems to other users (so e.g. *not* having a windowing<br>
protocol could actually be considered a feature, since it left some<br>
bandwidth available for the HTTP requests to get out).<br>
<br>
That said, I can think of a couple of likely slowdowns.<br>
<br>
1: The process of uploading an immutable file involves a *lot* of<br>
roundtrips, which will hurt small files a lot more than large ones. Peer<br>
selection is done serially: we ask server #1, wait for a response, then<br>
ask server #2, wait, etc. This could be done in parallel, to a certain<br>
extent (it requires being somewhat optimistic about the responses you're<br>
going to get, and code to recover when you guess wrong). We then send<br>
share data and hashes in a bunch of separate messages (there is some<br>
overhead per message, but 1.5.0 has code to pipeline them [see #392], so<br>
I don't think this will introduce a significant number of roundtrips). I<br>
think that a small file (1kb) can be uploaded in peer-selection plus two<br>
RTT (one for a big batch of write() messages, the second for the final<br>
close() message), but the peer-selection phase will take a minimum of 10<br>
RTT (one per server contacted).<br>
<br>
I've been contemplating a rewrite of the uploader code for a while now.<br>
The main goals would be:<br>
<br>
* improve behavior in the repair case, where there are already shares<br>
present. Currently repair will blindly place multiple shares on the<br>
same server, and wind up with multiple copies of the same share.<br>
Instead, once upload notices any evidence of existing shares, it<br>
should query lots of servers to find all the existing shares, and<br>
then generate the missing ones and put them on servers that don't<br>
already have any. (#610, #362, #699)<br>
* improve tolerance to servers which disappear silently during upload,<br>
eventually giving up on the server instead of stalling for 10-30<br>
minutes<br>
* improve parallelism during peer-selection<br>
<br>
2: Foolscap is doing an awful lot of work to serialize and deserialize<br>
the data down at the wire level. This part of the work is roughly<br>
proportional to the number of objects being transmitted (so a list of<br>
100 32-byte hashes is a lot slower than a single 3200-byte string).<br>
<a href="http://foolscap.lothar.com/trac/ticket/117" target="_blank">http://foolscap.lothar.com/trac/ticket/117</a> has some notes on how we<br>
might speed up Foolscap. And the time spent in foolscap gets added to<br>
your round-trip times (we can't parallelize over that time), so it gets<br>
multiplied by the 12ish RTTs per upload. I suspect the receiving side is<br>
slower than the sending side, which means the uploader will spend a lot<br>
of time twiddling its thumbs while the servers think hard about the<br>
bytes they've just received.<br>
<br>
We're thinking about switching away from Foolscap for share-transfer and<br>
instead using something closer to HTTP (#510). This would be an<br>
opportunity to improve the RTT behavior as well: we don't really need to<br>
wait for an ACK before we send the next block, we just need confirmation<br>
that the whole share was received correctly, and we need to avoid<br>
buffering too much data in the outbound socket buffer. In addition, we<br>
could probably trim off an RTT by changing the semantics of the initial<br>
message, to combine a do-you-have-share query with a<br>
please-prepare-to-upload query. Or, we might decide to give up on<br>
grid-side convergence and stop doing the do-you-have-share query first,<br>
to speed up the must-upload case at the expense of the<br>
might-not-need-to-upload case. This involves a lot of other projects,<br>
though:<br>
<br>
* change the immutable share-transfer protocol to be less object-ish<br>
(allocate_buckets, bucket.write) and more send/more/more/done-ish.<br>
* change the mutable share design to make shares fully self-validating,<br>
with the storage-index as the mutable share pubkey (or its hash)<br>
* make the server responsible for validating shares as they arrive,<br>
replace the "write-enabler" with a rule that the server accepts a<br>
mutable delta if it results in a new share that validates against the<br>
pubkey and has a higher seqnum (this will also increase the server<br>
load a bit) (this is the biggest barrier to giving up Foolscap's<br>
link-level encryption)<br>
* replace the renew-lease/cancel-lease shared secrets with DSA pubkeys,<br>
again to tolerate the lack of link-level encryption<br>
* interact with Accounting: share uploads will need to be signed by a<br>
who-is-responsible-for-this-space key, and doing this over HTTP will<br>
be significantly different than doing it over foolscap.<br>
<br>
So this project is waiting for DSA-based lease management, new DSA-based<br>
mutable files, and some Accounting design work. All of these are waiting<br>
on getting ECDSA into pycryptopp.<br>
<br>
3: Using a nearby Helper might help and might hurt. You spend a lot more<br>
RTTs by using the helper (it effectively performs peer-selection twice,<br>
plus an extra couple RTTs to manage the client-to-helper negotiation),<br>
but now you're running encryption and FEC in separate processes, so if<br>
you have multiple cores or hyperthreading or whatever you can utilize<br>
the hardware better. The Helper (which is doing more foolscap messages<br>
per file than the client) is doing slightly less CPU work, so those<br>
message RTTs might be smaller, which might help. It would be an<br>
interesting exercise to graph throughput/latency against filesize using<br>
a local helper (probably on a separate machine but connected via LAN)<br>
and see where the break-even point is. We've got a ticket (#398) about<br>
having the client *not* use the Helper sometimes.. maybe a tahoe.cfg<br>
option to set a threshold filesize could be useful.<br>
<div class="im"><br>
> For example, if the local node supplies the SID to the helper, the<br>
> helper can do peer selection before the upload is completed.<br>
<br>
</div>Yeah, currently the Helper basically tries to locate shares, reports<br>
back the results of that attempt, then forgets everything it's learned<br>
and starts a new upload. We could probably preserve some information<br>
across that boundary to speed things up. Maybe the first query should be<br>
allocate_buckets(), and if the file was already in place, we abort the<br>
upload, but if it wasn't, we use the returned bucket references. That<br>
could shave off N*RTT. We should keep this in mind during the Uploader<br>
overhaul; it'll probably influence the design.<br>
<br>
cheers,<br>
<font color="#888888"> -Brian<br>
</font><div><div></div><div class="h5">_______________________________________________<br>
tahoe-dev mailing list<br>
<a href="mailto:tahoe-dev@allmydata.org">tahoe-dev@allmydata.org</a><br>
<a href="http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev" target="_blank">http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev</a><br>
</div></div></blockquote></div><br>