#3777 closed task (fixed)

Some potential issues with GET /v1/immutable/:storage_index

Reported by: itamarst Owned by: exarkun
Priority: normal Milestone: HTTP Storage Protocol
Component: unknown Version: n/a
Keywords: Cc:
Launchpad Bug:

Description (last modified by itamarst)

Downloading multiple shares is specified via the query string.

The current spec gives an example:

GET /v1/immutable/:storage_index?share=:s0&share=:sN&offset=o1&size=z0&offset=oN&size=zN

First, that example is clearly wrong (should be "offset=o0", not "offset=o1".).

Second, it's not made explicit how to match shares, offsets, and sizes. Implicitly it seems like the pattern is:

  1. First all the shares.
  2. Then, pairs of offset and size, in same order as shares.

This pattern should be explicitly documented.

Third, as part of documenting, it's worth thinking about the pattern in context of HTTP server implementations. HTTP server frameworks often don't preserve order _between_ parameters, only preserving order for multiple values of a single parameter. So e.g. ?share=1&share=2&offset=0&offset=100 will look the same as ?share=1&offset=0&share=2&offset=100 if you're using Twisted Web (or Flask), because you get back a mapping of argument key to list of values.

Fourth, can I request multiple chunks from same share?

Fifth, taking a step back and looking at the big picture, I am not certain that supporting reading from multiple shares (or more broadly, reading multiple chunks) in a single query is actually useful.

  1. It's not clear to me it simplifies the client implementation. Clients can just send multiple HTTP requests in parallel.
  2. There are Python API design difficulties of receive arbitrarily sized multiple chunks in a single stream.
  3. Using single CBOR result streaming isn't really easy to support. Alternative is to concatenate multiple separately CBOR-encoded values (i.e. writing multiple results of cbor2.dumpb or whatever), which CBOR libraries can handle better.
  4. Given support for parallel requests, it doesn't improve latency (though it does have minor reduction in bandwidth usage).

Change History (5)

comment:1 Changed at 2021-08-23T14:22:42Z by itamarst

  • Description modified (diff)
  • Milestone changed from undecided to HTTP Storage Protocol
  • Owner set to exarkun
  • Type changed from defect to task

comment:2 Changed at 2021-08-23T15:27:12Z by itamarst

All the above applies to mutables too, I guess.

comment:3 Changed at 2021-08-23T15:27:41Z by itamarst

  • Description modified (diff)

comment:5 Changed at 2021-09-07T21:52:11Z by GitHub <noreply@…>

  • Resolution set to fixed
  • Status changed from new to closed

In bec813f/trunk:

Merge pull request #1119 from LeastAuthority?/3777.gbs-immutable-download-spec-improvements

GBS immutable download spec improvements

Fixes: ticket:3777

Note: See TracTickets for help on using tickets.