#1110 new enhancement

pipeline download blocks for better performance

Reported by: zooko Owned by: nobody
Priority: major Milestone: soon
Component: code-network Version: 1.7.0
Keywords: download performance unfinished-business Cc: tahoe-lafs.org@…
Launchpad Bug:


As Brian and I have discussed in person, downloads would probably be a bit faster for some users if we pipelined requests for successive blocks. Brian and I casually agreed that a pipeline depth of 2 would probably be pretty good for lots of users.

Change History (9)

comment:1 Changed at 2010-08-05T21:29:25Z by zooko

  • Milestone changed from 1.8.0 to eventually

comment:2 Changed at 2010-08-05T21:38:07Z by zooko



This is a good example of how pipelining download of blocks (#1110) could help. Previously I thought of it as a performance improvement when downloading successive blocks of the same file. Therefore I figured that if you were doing streaming processing of the file, such as if it was a movie and you were playing it out at normal speed, then a sufficiently large segment size would make the download faster than your playout speed, so pipelining would not matter for that. But this example shows that for some cases the segment size is irrelevant—in this case (if Brian's guess is correct) a read-block-pipeline depth of >= 2 would take one round-trip off of the startup time.

comment:3 Changed at 2010-09-06T06:31:42Z by zooko

  • Owner changed from warner to nobody

comment:4 Changed at 2010-09-06T06:44:40Z by zooko


Kyle's benchmarks and discussion and links to code:

Intriguing! It looks like upload typically took about 150 seconds and download took at least 850! Upload has pipelining and download doesn't. I wonder if that could account for *all* of that large difference!

This is issue #1110. It would probably make an excellent first hack for a new Tahoe-LAFS coder in the v1.9 timeframe. :-)

comment:5 Changed at 2010-09-07T00:44:37Z by davidsarah

#1187 is a more ambitious generalization of this ticket. If you pipeline successive shares, but still download a fixed set of shares per segment per server, then the potential gain is limited by the fact that for each segment, you still have to wait for the server that finishes last. What #1187 proposes would tend to keep the pipe from each server as full as possible, by downloading as many shares from each server as bandwidth allows.

It may be worth doing this ticket first, but with an eye to how to extend it.

comment:6 Changed at 2010-09-07T16:10:09Z by zooko

  • Keywords unfinished-business added
  • Milestone changed from eventually to 1.9.0

Marking this as 1.9.0 and "unfinished-business" as mentioned in http://tahoe-lafs.org/pipermail/tahoe-dev/2010-September/005163.html .

comment:7 Changed at 2011-07-27T18:22:37Z by zooko

  • Milestone changed from 1.9.0 to soon

comment:8 Changed at 2014-03-02T13:50:21Z by daira

Eek, I'm shocked this isn't fixed yet.

comment:9 Changed at 2016-07-11T16:47:11Z by lpirl

  • Cc tahoe-lafs.org@… added
Note: See TracTickets for help on using tickets.