#1186 closed enhancement (duplicate)

downloader: avoid redundant Share.loop calls

Reported by: warner Owned by:
Priority: major Milestone: 1.9.0
Component: code-encoding Version: 1.8β
Keywords: performance Cc:
Launchpad Bug:


In #1170 (c92) I mentioned a performance problem in which the downloader (specifically Share._got_response) queues a new call to Share.loop for each message received (which is linear with the number of segments, roughly k*3*numsegs total). Given the way that messages arrive in the pipeline, most of these calls have no work to do, and each one takes about 250us on my laptop. In a purely local download (running at 4.4MBps), this wastes about 8% of the download time.

The fix is to set a flag when the loop needs to be run, queue a call if-and-only-iff the flag was not already set, and clear the flag when the loop is run.

Change History (2)

comment:1 Changed at 2010-09-06T13:39:31Z by zooko

I think you mean ShareFinder._got_response instead of Share._got_response. Let's see, how does it trigger ShareFinder.loop? It calls ShareFinder._deliver_shares which eventually calls DownloadNode.got_shares which calls SegmentFetcher.add_shares which eventually calls SegmentFetcher.loop.

comment:2 Changed at 2011-09-15T16:20:18Z by warner

  • Milestone changed from soon to 1.9.0
  • Resolution set to duplicate
  • Status changed from new to closed

Nope, ShareFinder does the DYHB queries, so it's only involved once at the beginning of the download. It looks like I really meant Share._got_data, which is called multiple times per fetched block: one for the block data itself, another couple times for hashes, made worse by the fact that we're doing independent (although pipelined) foolscap remote_read()s, since we don't have an immutable readv storage API.

But it turns out that this is a dup: #1268 is the real ticket, and it was fixed in c9def7697757bdae.

Note: See TracTickets for help on using tickets.