[tahoe-lafs-trac-stream] [tahoe-lafs] #1288: support streaming uploads in uploader

[tahoe-lafs]

#1288: support streaming uploads in uploader
-------------------------+-------------------------------------------------
     Reporter:           |      Owner:
  davidsarah             |     Status:  new
         Type:           |  Milestone:  undecided
  enhancement            |    Version:  1.8.1
     Priority:  major    |   Keywords:  streaming performance upload sftp
    Component:  code-    |  fuse reliability
  encoding               |
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------

Comment (by davidsarah):

 Replying to [comment:1 zooko]:
 > If we wanted to implement this ticket and still to support client-side
 deduplication, which saves upload bandwidth and server-side storage space,
 then we'd have to make it be an option. For this upload do you want to
 make a pass over the data first, to see if it is already stored and you
 might be able to skip the upload, or do you want to do a streaming upload,
 where the storage client (== Tahoe-LAFS gateway) does not have to store
 temporary copy of the entire file in order to make two passes over it?

 Well, another possibility is that the client starts to upload the file,
 but aborts the upload if it finishes making a pass over the data and
 detects that it was already stored. That might make sense if the client is
 receiving the file faster than it is able to upload it.

 A difficulty here is that without knowing the file's hash, the client
 can't determine the optimum set of servers to store shares on. But if the
 number of servers on the grid were not much greater than
 {{{shares.total}}}, then that might not matter, because it could start
 uploading shares to all servers. (Or there could be some cleverer way to
 work around this problem that I'm not seeing right now.)

-- 
Ticket URL: <http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1288#comment:2>
tahoe-lafs <http://tahoe-lafs.org>
secure decentralized storage