[tahoe-dev] [tahoe-lafs] #1170: new-downloader performs badly when downloading a lot of data from a file

[tahoe-lafs]

#1170: new-downloader performs badly when downloading a lot of data from a file
------------------------------+---------------------------------------------
     Reporter:  zooko         |       Owner:                                           
         Type:  defect        |      Status:  new                                      
     Priority:  critical      |   Milestone:  1.8.0                                    
    Component:  code-network  |     Version:  1.8β                                     
   Resolution:                |    Keywords:  immutable download performance regression
Launchpad Bug:                |  
------------------------------+---------------------------------------------

Comment (by warner):

 BTW, be sure to pay attention to the {{{DataSpans}}} too, specifically
 {{{Share._received}}} . That's the one that I observed growing linearly
 with
 number-of-segments-read.

 I'm close to finishing my rework of the way Shares are handled. If we can
 make new-downloader fast enough by fixing complexity issues in spans.py,
 we
 should stick with that for 1.8.0, because those are probably smaller and
 less
 intrusive changes. If not, here are the properties of my Share-handling
 changes:

  * use a new diversity-seeking Share selection algorithm, as described in
    comment:27 . This should distribute the download load evenly among all
    known servers when they have equal number of shares, and as evenly as
    possible (while still getting k shares) when not. If more shares are
    discovered later, the algorithm will recalculate the sharemap and take
    advantage of the new shares, and we'll keep looking for new shares as
 long
    as we don't have the diversity that we want (one share per server).

  * fix the problem in which late shares (not used for the first segment,
 but
    located and added later) were not given the right sized hashtree and
 threw
    errors, causing them to be dropped. I think this completely broke the
    "tolerate loss of servers" feature, but the problem might have been
 caused
    by the diversity-seeking algorithm change, rather than something that
 was
    in new-downloader originally.

  * deliver all shares to the {{{SegmentFetcher}}} as soon as we learn
 about
    them, instead of waiting for the fetcher to tell us it's hungry. This
    gives the fetcher more information to work with.

 I might be able to attach a patch tomorrow.. there are still some bugs in
 it,
 and I haven't finished implementing the last point (push shares on
 discovery,
 not pull on hunger).

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1170#comment:53>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage