[tahoe-dev] [tahoe-lafs] #1170: new-downloader performs badly when downloading a lot of data from a file
tahoe-lafs
trac at tahoe-lafs.org
Wed Aug 18 18:56:04 UTC 2010
#1170: new-downloader performs badly when downloading a lot of data from a file
------------------------------+---------------------------------------------
Reporter: zooko | Owner:
Type: defect | Status: new
Priority: critical | Milestone: 1.8.0
Component: code-network | Version: 1.8β
Resolution: | Keywords: immutable download performance regression
Launchpad Bug: |
------------------------------+---------------------------------------------
Comment (by warner):
BTW, be sure to pay attention to the {{{DataSpans}}} too, specifically
{{{Share._received}}} . That's the one that I observed growing linearly
with
number-of-segments-read.
I'm close to finishing my rework of the way Shares are handled. If we can
make new-downloader fast enough by fixing complexity issues in spans.py,
we
should stick with that for 1.8.0, because those are probably smaller and
less
intrusive changes. If not, here are the properties of my Share-handling
changes:
* use a new diversity-seeking Share selection algorithm, as described in
comment:27 . This should distribute the download load evenly among all
known servers when they have equal number of shares, and as evenly as
possible (while still getting k shares) when not. If more shares are
discovered later, the algorithm will recalculate the sharemap and take
advantage of the new shares, and we'll keep looking for new shares as
long
as we don't have the diversity that we want (one share per server).
* fix the problem in which late shares (not used for the first segment,
but
located and added later) were not given the right sized hashtree and
threw
errors, causing them to be dropped. I think this completely broke the
"tolerate loss of servers" feature, but the problem might have been
caused
by the diversity-seeking algorithm change, rather than something that
was
in new-downloader originally.
* deliver all shares to the {{{SegmentFetcher}}} as soon as we learn
about
them, instead of waiting for the fetcher to tell us it's hungry. This
gives the fetcher more information to work with.
I might be able to attach a patch tomorrow.. there are still some bugs in
it,
and I haven't finished implementing the last point (push shares on
discovery,
not pull on hunger).
--
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1170#comment:53>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage
More information about the tahoe-dev
mailing list