[tahoe-lafs-trac-stream] [tahoe-lafs] #1288: support streaming uploads in uploader

[tahoe-lafs]

#1288: support streaming uploads in uploader
-------------------------+-------------------------------------------------
     Reporter:           |      Owner:
  davidsarah             |     Status:  new
         Type:           |  Milestone:  undecided
  enhancement            |    Version:  1.8.1
     Priority:  major    |   Keywords:  streaming performance upload sftp
    Component:  code-    |  fuse reliability
  encoding               |
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------

Comment (by zooko):

 Note that doing a streaming upload—where the storage servers are accepting
 and storing the first blocks of your file from you before you (the storage
 client) have even looked at the last blocks of that file—is inherently
 incompatible with client-side deduplication—where you realize that the
 file is already stored before you upload the first block.

 If we wanted to implement this ticket and still to support client-side
 deduplication, which saves upload bandwidth and server-side storage space,
 then we'd have to make it be an option. For this upload do you want to
 make a pass over the data first, to see if it is already stored and you
 might be able to skip the upload, or do you want to do a streaming upload,
 where the storage client (== Tahoe-LAFS gateway) does not have to store
 temporary copy of the entire file in order to make two passes over it?

 A streaming upload could be compatible with server-side deduplication,
 where after the last block of the share is uploaded, the server says "Oh
 look, I already have a copy of this share. I'll just delete the new one
 and add a new lease to the old one.". This doesn't help with upload
 bandwidth but conserves server-side storage space.

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1288#comment:1>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage