[tahoe-dev] [tahoe-lafs] #994: support precompressed files

[tahoe-lafs]

#994: support precompressed files
--------------------------------------------+-------------------------------
 Reporter:  davidsarah                      |           Owner:  somebody 
     Type:  enhancement                     |          Status:  new      
 Priority:  major                           |       Milestone:  undecided
Component:  code                            |         Version:  1.6.0    
 Keywords:  compress performance integrity  |   Launchpad_bug:           
--------------------------------------------+-------------------------------
 A "precompressed file" is a file where the plaintext has been compressed
 using an algorithm supported by HTTP (gzip or deflate -- we'd probably
 support only one). When the file is served via the webapi, it is served in
 compressed form with the [http://tools.ietf.org/html/rfc2616#section-14.11
 Content-Encoding] HTTP header set appropriately. The Content-Encoding can
 also be set in a PUT or POST request to upload a precompressed file.

 Storage servers would be completely ignorant of precompressed files. The
 CLI, SFTP and FTP frontends would have to be decompress them. The gateway
 would also have to decompress if it receives an HTTP request that does not
 have an [http://tools.ietf.org/html/rfc2616#section-14.3 Accept-Encoding]
 header allowing the compression algorithm used for that file.

 This would provide a performance improvement as long as the HTTP clients
 have sufficient CPU capacity, that the time taken for them to decompress
 is outweighed by the savings in bandwidth. CPU-constrained clients
 (connecting to a less CPU-constrained gateway) are not a problem because
 they can just not set {{{Accept-Encoding}}}.

 This would rely on HTTP clients implementing decompression correctly; if
 they don't then there is a potential loss of integrity, and the
 possibility of attacks against the client from maliciously constructed
 compressed data. It is possible to protect against "decompression bombs"
 if that is required.

-- 
Ticket URL: <http://allmydata.org/trac/tahoe/ticket/994>
tahoe-lafs <http://allmydata.org>
secure decentralized file storage grid