#3946 closed enhancement (fixed)

Reduce HTTP protocol chattiness for downloads

Reported by: itamarst Owned by: GitHub <noreply@…>
Priority: normal Milestone: HTTP Storage Protocol
Component: unknown Version: n/a
Keywords: Cc:
Launchpad Bug:

Description

#3939 reduced upload chattiness a lot; there is still likely many improvements that could be done when downloading.

Change History (7)

comment:1 Changed at 2022-12-05T19:25:02Z by itamarst

For immutables, allmydata/immutable/downloader/share.py has a bunch of code that devolves to downloading blocks.

comment:2 Changed at 2022-12-13T17:00:47Z by itamarst

And there's a comment: # TODO: quantize to reasonably-large blocks YES YES YOU SHOULD

comment:3 Changed at 2023-01-23T17:02:08Z by itamarst

Increasing segment size would help, but:

comment:4 Changed at 2023-01-24T15:15:44Z by itamarst

The code with the quantize path is maybe not what clients use? Investigating.

Version 0, edited at 2023-01-24T15:15:44Z by itamarst (next)

comment:5 Changed at 2023-01-24T16:30:27Z by itamarst

The segment fetching code is complex enough that I don't think it's worth messing with at this juncture. So I'm just going to increase segment size, which at least will help with new uploads. Let's say... 1MB, from current 128KB.

Required testing and checks:

  • An upload with 128KB segment can be downloaded by client with 1MB expected max segment.
  • An upload with 1MB segment can be downloaded by client with 128KB expected max segment.
  • Validate all uses of DEFAULT_MAX_SEGMENT_SIZE are advisory, if not add more bullet points here.

comment:6 Changed at 2023-01-24T16:41:55Z by itamarst

With 40ms round-trip latency, and downloading a 20MB immutable file:

  • 128KB segment size means a 15 second download.
  • 1MB segment size means a 3.5 second download.

So that's still kinda terrible (there's no bandwidth bottleneck in my test setup!) but much much better.

Last edited at 2023-01-24T16:42:11Z by itamarst (previous) (diff)

comment:7 Changed at 2023-02-21T14:37:35Z by GitHub <noreply@…>

  • Owner set to GitHub <noreply@…>
  • Resolution set to fixed
  • Status changed from new to closed

In 0b0af62/trunk:

Merge pull request #1251 from tahoe-lafs/3946-less-chatty-downloads

Make immutable downloads less chatty

Fixes ticket:3946

Note: See TracTickets for help on using tickets.