Opened at 2009-09-24T06:16:06Z

Last modified at 2011-08-18T21:00:28Z

#809 new enhancement

Measure how segment size affects upload/download speed.

Reported by: zooko Owned by: warner
Priority: major Milestone: soon
Component: code-network Version: 1.5.0
Keywords: easy performance upload download statistics Cc:
Launchpad Bug:

Description

#392 has measurements from Brian showing that reducing segment size from 1 MiB to 128 KiB reduced upload/download speed drastically on a high-speed LAN. See if this is still true. Maybe make a little graph of upload/download speed at different segments sizes. There is a script being run by the buildbot to measure upload/download speed, which you could use:

​http://allmydata.org/buildbot/builders/speed-colo

Change History (13)

comment:1 Changed at 2009-12-04T18:48:59Z by zooko

If you love this ticket, you might also like #398 (allow users to disable use of helper: direct uploads might be faster) and #320 (add streaming (on-line) upload to HTTP interface).

comment:2 Changed at 2009-12-12T20:46:21Z by davidsarah

  • Keywords performance added

comment:3 Changed at 2009-12-12T20:57:36Z by davidsarah

  • Keywords upload download added

comment:4 Changed at 2010-02-27T06:41:46Z by zooko

  • Milestone changed from undecided to 1.7.0

comment:5 Changed at 2010-02-27T06:45:44Z by zooko

  • Owner set to warner

comment:6 Changed at 2010-05-08T19:44:54Z by zooko

If you like this ticket, you might also like the "Brian's New Downloader" bundle of tickets: #800 (improve alacrity by downloading only the part of the Merkle Tree that you need), #798 (improve random-access download to retrieve/decrypt less data), #448 (download: speak to as few servers as possible), #287 (download: tolerate lost or missing servers), and #605 (two-hour delay to connect to a grid from Win32, if there are many storage servers unreachable).

comment:7 Changed at 2010-05-08T22:46:18Z by zooko

  • Milestone changed from 1.7.0 to 1.8.0

Brian's New Downloader is now planned for v1.8.0.

comment:8 Changed at 2010-05-16T05:28:48Z by zooko

By the way Brian has big plans to make some sort of beautiful graphical visualization of network behavior/performance. I can't wait! For one thing, I intend to work on upload code in v1.8 and such visualization would help tremendously.

comment:9 Changed at 2010-06-12T23:46:06Z by davidsarah

  • Keywords statistics added

comment:10 Changed at 2010-08-12T20:59:18Z by zooko

Kyle Markley did some good work on this but the results are inconclusive:

More measurements and more analysis of Kyle's measurements (especially his packet traces) is warranted!

See also Nathan Eisenberg's benchmarks of 1.7.1 (standard pipeline and segment size settings) with 1.8.0c1 (standard pipeline and segment size settings):

comment:11 Changed at 2010-08-12T20:59:31Z by zooko

  • Milestone changed from 1.8.0 to soon

comment:12 Changed at 2010-09-14T02:03:29Z by davidsarah

If #1110 or #1187 (download pipelining) were implemented, then the download performance should in theory be less sensitive to segment size. The segment size could then be reduced to improve memory usage.

comment:13 Changed at 2011-08-18T21:00:28Z by zooko

Trunk now has the new visualizer in it, so you could run downloads with different segment sizes and then look at the visualizations to see how the segments sizes changed the network behavior.

Note: See TracTickets for help on using tickets.