[tahoe-dev] [tahoe-lafs] #1170: new-downloader performs badly when downloading a lot of data from a file

[tahoe-lafs]

#1170: new-downloader performs badly when downloading a lot of data from a file
------------------------------+---------------------------------------------
     Reporter:  zooko         |       Owner:                                           
         Type:  defect        |      Status:  new                                      
     Priority:  critical      |   Milestone:  1.8.0                                    
    Component:  code-network  |     Version:  1.8β                                     
   Resolution:                |    Keywords:  immutable download performance regression
Launchpad Bug:                |  
------------------------------+---------------------------------------------

Comment (by warner):

 Replying to [comment:61 zooko]:

 > Hm, hey waitasecond, in my earlier testing (recorded in this ticket),
 > 1.8.0c2 was ''faster'' then 1.7.1 for small files (<= 10 MB). This was
 > also the case for Nathan Eisenberg's benchmarks (posted to tahoe-dev).
 > But currently it looks to me like the average download speed (as
 > reported by curl during its operation) is the same at the beginning of
 > the download as at the end, i.e. even during the first 10 MB or so
 > 1.8.0c2 is only getting about 150 KBps where 1.7.1 is getting more
 > than 200 KBps. Did something change?

 There's a sizeable startup time in 1.7.1 (lots of roundtrips), which
 went away in 1.8.0c2 . I think we're all in agreement about the
 small-file speedups that provides (i.e. we've not seen any evidence to
 the contrary). The change is on the order of a few seconds, though, so I
 think a 10MB file (or portion of a file) that takes 10MB/150kBps= 60s to
 complete won't be affected very much. I don't think you'll be able to
 see its effects in the curl output.

 Nathan's tests were on hundreds or thousands of small files.

 From my tests, the new-downloader sees about 500ms more taken to
 complete the first segment than the second and following ones. I believe
 that's the time spend doing server selection, UEB fetches, and the large
 hash chain fetches.

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1170#comment:67>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage