[tahoe-dev] [tahoe-lafs] #757: there isn't a doc that says "which operations are efficient"

[tahoe-lafs]

#757: there isn't a doc that says "which operations are efficient"
--------------------------------------------------+-------------------------
 Reporter:  zooko                                 |           Owner:  nobody   
     Type:  enhancement                           |          Status:  new      
 Priority:  major                                 |       Milestone:  undecided
Component:  documentation                         |         Version:  1.4.1    
 Keywords:  performance docs large review-needed  |   Launchpad_bug:           
--------------------------------------------------+-------------------------

Comment(by zooko):

 There is no way currently in the cli, wui or the wapi to upload a file
 ''without'' using convergent encryption. (The confidentiality risk of
 convergent encryption is solved by adding in a separate "added convergence
 secret", not by skipping the step of hashing the cleartext to generate a
 symmetric key.) Therefore, all uploads of immutable files take two passes
 over the file.  If you're uploading through the wui/wapi, this means your
 client (i.e. web browser, or a wapi-using client) first reads the entire
 file from disk while streaming it to the gateway, then the gateway writes
 it out to a temporary directory on disk while hashing it to generate the
 symmetric encryption key, then the gateway reads it again from the
 beginning from its temporary location on disk while encrypting it,
 erasure-coding it, and uploading the shares to the storage servers.

 If you're uploading from within the tahoe node process itself (i.e. you've
 extended your tahoe node with your own code instead of using the wapi)
 then it will make two consecutive passes of reading the entire file from
 its original location on disk and then encrypt, erasure-code, and upload
 during the second pass.

 #329 (add streaming (on-line) upload to HTTP interface) is about allowing
 one-pass "streaming" upload, so for example the web gateway would no
 longer write a temporary copy of the file to disk at all but would instead
 process it incrementally in (a small amount of) RAM. Shawn Willden
 contributed the first step of #320, which is code to use a random
 encryption key instead of to hash the file and generate a convergent
 encryption key. See:
 [source:src/allmydata/immutable/upload.py at 4164#L1099]. However, as far as
 I can tell from the wapi code and the docs, there is no way to access this
 feature through the wapi. (I thought that Shawn contributed a patch to
 make this feature available. Maybe that patch never got accepted into
 trunk?)

-- 
Ticket URL: <http://allmydata.org/trac/tahoe/ticket/757#comment:18>
tahoe-lafs <http://allmydata.org>
secure decentralized file storage grid