[tahoe-lafs-trac-stream] [tahoe-lafs] #1571: S3 backend: support streaming reads of immutable and mutable shares

[tahoe-lafs]

#1571: S3 backend: support streaming reads of immutable and mutable shares
-------------------------+-------------------------------------------------
     Reporter:           |      Owner:
  davidsarah             |     Status:  new
         Type:  defect   |  Milestone:  undecided
     Priority:  major    |    Version:  1.9.0b1
    Component:  code-    |   Keywords:  streaming performance memory s3
  storage                |  cloud-backend storage
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------
Changes (by daira):

 * keywords:  streaming performance memory s3-backend storage => streaming
     performance memory s3 cloud-backend storage


Old description:

> The current S3 backend implementation waits until a GET request to S3 for
> the share has completed before creating a share object that can be read.
> This ticket is to make it support reading as much data as has been
> received before the GET is complete. Note that the SFTP frontend does a
> similar thing in {{{OverwriteableFileConsumer}}}; some of that code or
> the approach it uses may be reusable.
>
> In the case of mutable shares, the client would need to use multiple
> {{{remote_readv}}} calls with relatively short lengths. This ticket is
> not about that, only about whether the S3 backend would support such
> usage efficiently.
>
> This ticket might need to be split for mutable and immutable shares, but
> I'll leave it as one for the time being.
>
> The security issue in #1570 doesn't apply here; we are not relying on the
> S3 authentication, since the shares are encrypted.

New description:

 The current S3 backend implementation waits until a GET request to S3 for
 the share has completed before creating a share object that can be read.
 This ticket is to make it support reading as much data as has been
 received before the GET is complete. Note that the SFTP frontend does a
 similar thing in {{{OverwriteableFileConsumer}}}; some of that code or the
 approach it uses may be reusable.

 In the case of mutable shares, the client would need to use multiple
 {{{remote_readv}}} calls with relatively short lengths. This ticket is not
 about that, only about whether the S3 backend would support such usage
 efficiently.

 This ticket might need to be split for mutable and immutable shares, but
 I'll leave it as one for the time being.

 The security issue in #1570 doesn't apply here; we are not relying on the
 S3 authentication, since the shares are encrypted.

--

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1571#comment:2>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage