#1571 closed defect (wontfix)

S3 backend: support streaming reads of immutable and mutable shares

Reported by: davidsarah Owned by:
Priority: major Milestone: undecided
Component: code-storage Version: 1.9.0b1
Keywords: streaming performance memory s3 cloud-backend storage Cc:
Launchpad Bug:

Description (last modified by daira)

The current S3 backend implementation waits until a GET request to S3 for the share has completed before creating a share object that can be read. This ticket is to make it support reading as much data as has been received before the GET is complete. Note that the SFTP frontend does a similar thing in OverwriteableFileConsumer; some of that code or the approach it uses may be reusable.

In the case of mutable shares, the client would need to use multiple remote_readv calls with relatively short lengths. This ticket is not about that, only about whether the S3 backend would support such usage efficiently.

This ticket might need to be split for mutable and immutable shares, but I'll leave it as one for the time being.

The security issue in #1570 doesn't apply here; we are not relying on the S3 authentication, since the shares are encrypted.

Change History (3)

comment:1 Changed at 2011-10-25T13:00:47Z by davidsarah

  • Description modified (diff)

comment:2 Changed at 2014-03-18T18:12:48Z by daira

  • Description modified (diff)
  • Keywords s3 cloud-backend added; s3-backend removed

comment:3 Changed at 2020-10-30T12:35:44Z by exarkun

  • Resolution set to wontfix
  • Status changed from new to closed

The established line of development on the "cloud backend" branch has been abandoned. This ticket is being closed as part of a batch-ticket cleanup for "cloud backend"-related tickets.

If this is a bug, it is probably genuinely no longer relevant. The "cloud backend" branch is too large and unwieldy to ever be merged into the main line of development (particularly now that the Python 3 porting effort is significantly underway).

If this is a feature, it may be relevant to some future efforts - if they are sufficiently similar to the "cloud backend" effort - but I am still closing it because there are no immediate plans for a new development effort in such a direction.

Tickets related to the "leasedb" are included in this set because the "leasedb" code is in the "cloud backend" branch and fairly well intertwined with the "cloud backend". If there is interest in lease implementation change at some future time then that effort will essentially have to be restarted as well.

Note: See TracTickets for help on using tickets.