[tahoe-lafs-trac-stream] [tahoe-lafs] #1545: add readv() API to immutable-share storage-server protocol, use in downloader

[tahoe-lafs]

#1545: add readv() API to immutable-share storage-server protocol, use in
downloader
--------------------------+---------------------------
 Reporter:  warner        |          Owner:
     Type:  enhancement   |         Status:  new
 Priority:  major         |      Milestone:  undecided
Component:  code-storage  |        Version:  1.9.0a2
 Keywords:  performance   |  Launchpad Bug:
--------------------------+---------------------------
 One of the most obvious fixes for the immutable-download performance
 problems tracked in #1264 (and on the [[Performance/Sep2011]] results) is
 to implement a scatter/gather {{{readv()}}} method for immutable shares.
 The graphs show MDMF downloads running just as fast with k=60 as with k=3,
 whereas for immutable files there is a drastic slowdown (10x) between k=3
 and k=60. We're still investigating, but I suspect that Foolscap's
 message-serialization performance is to blame, and an easy way to mitigate
 that is to send fewer messages.

 The interface should probably be just like the mutable-share's
 {{{remote_readv()}}} API: a vector of {{{(offset,length)}}} tuples, and
 the return value is a vector of data strings. (A future HTTP-based
 interface will probably pack these vectors into a single string, but we
 might experiment with doing that here too (basically do the marshalling
 before handing anything to foolscap, trading off generality for
 performance).

 David-Sarah mentioned that some of their new storage-backend code (for
 LAE) provides this interface, so we're likely to have the back half of
 this feature fairly soon. The rest of the work is to change
 immutable/downloader/share.py to turn a Request span into a read vector,
 instead of looping over all pieces of the span and sending separate
 {{{read()}}} requests for each.

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1545>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage