#1545 assigned enhancement

add readv() API to immutable-share storage-server protocol, use in downloader

Reported by: warner Owned by: davidsarah
Priority: major Milestone: soon
Component: code-storage Version: 1.9.0a2
Keywords: performance Cc:
Launchpad Bug:


One of the most obvious fixes for the immutable-download performance problems tracked in #1264 (and on the Performance/Sep2011 results) is to implement a scatter/gather readv() method for immutable shares. The graphs show MDMF downloads running just as fast with k=60 as with k=3, whereas for immutable files there is a drastic slowdown (10x) between k=3 and k=60. We're still investigating, but I suspect that Foolscap's message-serialization performance is to blame, and an easy way to mitigate that is to send fewer messages.

The interface should probably be just like the mutable-share's remote_readv() API: a vector of (offset,length) tuples, and the return value is a vector of data strings. (A future HTTP-based interface will probably pack these vectors into a single string, but we might experiment with doing that here too (basically do the marshalling before handing anything to foolscap, trading off generality for performance).

David-Sarah mentioned that some of their new storage-backend code (for LAE) provides this interface, so we're likely to have the back half of this feature fairly soon. The rest of the work is to change immutable/downloader/share.py to turn a Request span into a read vector, instead of looping over all pieces of the span and sending separate read() requests for each.

Attachments (1)

readv.diff (8.0 KB) - added by warner at 2011-09-27T06:54:45Z.
add readv() support to server, use it from the client if available

Download all attachments as: .zip

Change History (4)

comment:1 Changed at 2011-09-27T06:53:43Z by warner

early results suggest that doing this would speed up high-k immutable downloads by about 24%. For example, k=54 with trunk takes roughly 414s to download a 100MB file (6 servers, 3 hosts, LAN connections). When basic readv() is used, this drops to 317s. For small k (like the default k=3), the effect is less clear, however there still seems to be a significant improvement (k=3 trunk 100MB takes maybe 38s, with-readv takes 32s).

The effect is roughly halfway between unmodified trunk CHK and trunk MDMF (which prefetches the whole block_hash_tree and doesn't even have a crypttext_hash_tree).

Changed at 2011-09-27T06:54:45Z by warner

add readv() support to server, use it from the client if available

comment:2 Changed at 2011-09-27T06:55:42Z by warner

That patch is just a proof-of-concept, not actually ready or recommended for landing. I'm attaching it so others can reproduce my results.

comment:3 Changed at 2012-04-01T23:21:04Z by davidsarah

  • Milestone changed from undecided to 1.11.0
  • Owner set to davidsarah
  • Status changed from new to assigned
Note: See TracTickets for help on using tickets.