[tahoe-lafs-trac-stream] [tahoe-lafs] #1570: S3 backend: support streaming writes to immutable shares

[tahoe-lafs]

#1570: S3 backend: support streaming writes to immutable shares
-------------------------------------------------+-------------------------
 Reporter:  davidsarah                           |          Owner:
     Type:  defect                               |         Status:  new
 Priority:  major                                |      Milestone:
Component:  code-storage                         |  undecided
 Keywords:  security anti-censorship streaming   |        Version:  1.9.0b1
  performance memory s3-backend storage          |  Launchpad Bug:
-------------------------------------------------+-------------------------
 For immutable shares, the current S3 backend implementation writes data to
 a {{{StringIO}}} on each call to {{{write_share_data}}} (corresponding to
 {{{remote_write}}}), and only PUTs the share to S3 when the share is
 closed (corresponding to {{{remote_close}}}). This increases latency, and
 requires memory usage in the storage server proportional to the size of
 the share.

 Sending the data over HTTP as it is received by calls to
 {{{remote_write}}} is not difficult in principle, although it isn't
 currently supported by txaws. However, how would we calculate the MD5 hash
 of the share file? The {{{Content-MD5}}} header is optional, but without
 it there is an easy replay attack against the S3 authentication, allowing
 an attacker to replace share files without knowing the S3 secret key.
 (They cannot break integrity, but they can arbitrarily delete data.) A
 possible workaround is to require TLS for PUTs to S3, so that the
 {{{Authentication}}} header is secret and not available for replay. Note
 that this doesn't require the storage server to be authenticated at the
 TLS level, but it would require validating Amazon's certificate (or that
 of an S3 lookalike services) in order to prevent MITM attacks.

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1570>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage