pipeline download blocks for better performance

[zooko]

As Brian and I have discussed in person, downloads would probably be a bit faster for some users if we pipelined requests for successive blocks. Brian and I casually agreed that a pipeline depth of 2 would probably be pretty good for lots of users.

[zooko]

​http://tahoe-lafs.org/pipermail/tahoe-dev/2010-August/004909.html

says:

This is a good example of how pipelining download of blocks (#1110) could help. Previously I thought of it as a performance improvement when downloading successive blocks of the same file. Therefore I figured that if you were doing streaming processing of the file, such as if it was a movie and you were playing it out at normal speed, then a sufficiently large segment size would make the download faster than your playout speed, so pipelining would not matter for that. But this example shows that for some cases the segment size is irrelevant—in this case (if Brian's guess is correct) a read-block-pipeline depth of >= 2 would take one round-trip off of the startup time.

[zooko]

​http://tahoe-lafs.org/pipermail/tahoe-dev/2010-September/005151.html

Kyle's benchmarks and discussion and links to code:

Intriguing! It looks like upload typically took about 150 seconds and download took at least 850! Upload has pipelining and download doesn't. I wonder if that could account for *all* of that large difference!

This is issue #1110. It would probably make an excellent first hack for a new Tahoe-LAFS coder in the v1.9 timeframe. :-)

[davidsarah]

#1187 is a more ambitious generalization of this ticket. If you pipeline successive shares, but still download a fixed set of shares per segment per server, then the potential gain is limited by the fact that for each segment, you still have to wait for the server that finishes last. What #1187 proposes would tend to keep the pipe from each server as full as possible, by downloading as many shares from each server as bandwidth allows.

It may be worth doing this ticket first, but with an eye to how to extend it.

[zooko]

Marking this as 1.9.0 and "unfinished-business" as mentioned in ​http://tahoe-lafs.org/pipermail/tahoe-dev/2010-September/005163.html .

[daira]

Eek, I'm shocked this isn't fixed yet.