[tahoe-dev] atlasgrid performance testing results

[Brian Warner]

I've spent the last week running some performance tests on trunk (which
will behave very much like the upcoming 1.9 release), using four
LAN-connected hosts generously donated by Atlas Networks.

The report is up on the wiki here:

 http://tahoe-lafs.org/trac/tahoe-lafs/wiki/Performance/Sep2011

The summary:

 * MDMF is nice and fast: 4MBps over LAN connections

 * my new CHK downloader (which appeared in 1.8.0) is not as fast
   (2.6MBps) as the old downloader (from 1.7.1) (4.4MBps)

 * high values of "k" really hurt the new CHK downloader (but not MDMF)

 * both CHK and MDMF do efficient partial reads (if you only read half
   the file, it takes half the time

This analysis reveals a number of fixes that we should experiment with.
The first big one is to implement a scatter/gather readv() method for
immutable-share downloads, to bring them more in line with mutable
retrievals. The more-experimental one is to play with plain HTTP access
for share download: the data suggests that Foolscap message
serialization is pretty expensive.

Tools and raw data are included on that wiki page.

Thanks again to Atlas for lending us thie grid: it makes doing this kind
of testing sooo much easier when you've got four quiet+dedicated
machines with fast network links to each other.

cheers,
 -Brian