implement the CHK-uploader helper, aka "offloaded uploader"

[warner]

I've been slowly working on an "offloaded uploader", in which the client node performs encryption but hands the crypttext off to an external node which does all the share encoding and uploading. This would allow us to offer a 1x upload service.

Most of the code split is already in place, there are a few more pieces needed to make it work.

My plan is to have a config file knob that provides the FURL of an uploading service, and if present, that service will be used for all uploaded files. It might also be useful to set a threshold of some sort: only offload files larger than 50kB, or something.

This will also raise some interesting questions about how we handle files that are already present in the grid: we'd like to avoid the 1x upload if the file is already present, so the uploading service should do peer selection (and thus discover the file is already present) before asking the client to send it any crypttext, in which case the upload can be terminated as successful.

[zooko]

Brian and I discussed this last week, and it seems like the majority of the users can be satisfied either by running a tahoe node locally or by using the current web api. Neither of those solutions are ideal on all dimensions, but they seem good enough to demote this ticket to the "undecided" Milestone.

[warner]

After some discussions with Peter, I'm thinking this is more important than zooko and I discussed the other day, so at least personally I'm going to raise its priority / bring it forward on my development schedule.

Independently of that, it occurred to me that we could build a form of the CLI tools that do the encryption, contact an encoding/decoding server, but which do not need to be long-running daemons. The 'tahoe ls' program would run just long enough to reach the desired directory, emit the results, then quit. This might make it more comfortable for someone to run these tools; they'd be more like utilities than a long-running application. There would be fewer opportunities for caching, of course, but we haven't really implemented much caching in the full tahoe node yet anyways. The fact that these clients only talk to a single server (the upload/download server) would make their short-livedness less costly than a node that needs to talk to dozens or hundreds of storage servers.

The Foolscap interface between these tools and the upload/download server should be in terms of a Storage Index and crypttext: submit an SI and get back a reference to an object that can do readv(), or one that can also do writev() and close().

As always, my goal is to allow users to keep their own secrets, in all of our use cases, and to make the choice between "keep my own secrets" and "please help me recover my password" modes be independent of anything else.

[zooko]

When you and I talked last week, we agreed that there was a substantial fraction of the user base who required to access their files through an unenhanced web browser, perhaps not even running on their own computer, and who required for us to remember their password for them, and that there was another important fraction who would run a node locally in order to have full control. We agreed that the remaining part of the user base that fell into neither category was a small enough fraction that we could put off serving them.

So what changed in your conversation with Peter or in your own thinking that counter-indicates this plan?

My personal preference is to encourage people in that remaining part of the user base to become part of the run-a-tahoe-node-locally part. Our company -- allmydata.com -- has been burned in the past by shipping an app that was written in Python which had critical performance problems, but tahoe doesn't have any such problems, so I hope that any users who say "I want to encrypt my data before sending it out of my own computer." will be satisfied with the answer of "No problem! Install this application and you're good to go.".

(Note that if they are already familiar with a particular user interface, they can continue to use that same user interface by connecting its back-end to ​http://127.0.0.1:8123 instead of whatever server they were previously using.)

[warner]

I'm thiking of the set of users who lie between those two groups: people who are able/willing/eager to run our code on their own home computers, but who balk at the 3x upload bandwidth that would be necessary to send shares instead of either plaintext or crypttext. For almost all home users (ADSL or cable modem), upload bandwidth is scarce, compared to download bandwidth. I think it will be easier to convince users that they should reap the security benefits of running their own node if they don't also have to suffer the hit of their uploads running several times slower.

I can't currently remember the specific issue that made me return to this point of view, but it occurred while I was talking to Peter about deployment scenarios, and what sort of backup app we should build on top of the code we have at this point. Perhaps Peter could chime in about how important/not this user set is?

[zooko]

Ah yes, upload bandwidth. That's important.

Also, Brian recently wrote:

"* implement the offloaded-uploader, and add an upload-server (aka "account server", aka "encoding server") FURL to the tahoe node's configuration. The benefits of this: 1x upload, reduced number of connections, avoiding duplicate uploads of files which were uploaded by other users, enabling resumeable uploads, improving simultaneous upload behavior."

It occured to me that if we have a Repairer, and if the Repairer can change the erasure coding parameters, then you could implement 1X upload by doing a p2p upload and then asking a repairer to repair it with higher expansion. The advantage of doing it this way would simply be that we already require p2p upload (for the grid backend), and we already require a repairer, so if the combination of the two also satisfies this other need, then that's less code we have to write and maintain.

Here is the list of benefits of the offloaded-uploader:

• 1X upload

I agree this is very important, and I'm not sure if my proposal of doing it with p2p upload with low overhead would suffice. I would like to measure the performance of that option before making up my mind.

• reduced number of connections

Could you say a few words about why you value this? I've often noticed in the pas that you value this, and you know a lot about TCP, so you probably have a good reason, but in my experience having multiple TCP connections isn't an issue. It certainly seems to work okay for BitTorrent?, for example.

• avoiding duplicate uploads of files which were uploaded by other users
• enabling resumeable uploads
• improving simultaneous upload behavior

Aren't these features more or less independent of the off-loaded uploader?

It would be valuable for the p2p-upload (the "backend") to have these features, too (and it already has the first of the three). Are we going to implement each of these features twice, or allow one or the other transfer mechanism to do without them?

The reason I'm pushing back on this so much is simply code size. I have the feeling that Tahoe is reaching the point where the inertia of the codebase limits the pace of development. Adding a new feature seems to take something like O(N) development time, where N is the number of current features.

If we could get by with using the same code for more use cases, this would be a big win.

[warner]

marking as 0.9.0 because it provides the goals set forth in the 0.9.0 milestone description (resumeable uploads), still much to discuss about this

[warner]

you could implement 1X upload by doing a p2p upload and then asking a repairer to repair it with higher expansion.

Yeah, that's a neat trick. It requires a dedicated colo-side Repairer, of course, but that's pretty much the same hassle as having colo-side upload helpers. A back-of-the-envelope calculation of performance could be done by counting the network io and disk io that must be done:

• offloaded-uploader approach:
  • client:
    • 1x disk read, 1x upstream network
  • helper:
    • streaming: 1x inbound network, 3x (k=3) outbound network
    • save-to-disk-first: same, plus maybe 1x disk read, 1x disk write
      • (save-to-disk-first enables resuming an interrupted upload. For

small files, the kernel's disk cache would avoid the actual IO)

• storage servers:
  • 3x inbound network, 3x disk writes (spread across all servers)

• 1x upload then repair:
  • client:
    • 1x disk read, 1x upstream network
  • storage servers (upload phase)
    • 1x inbound network, 1x disk write
  • repairer:
    • (naive) 1x inbound network, 3x outbound network (streaming)
    • (smarter, avoid sending shares back to their home) 1x inbound, 2x outbound
  • storage servers (repair phase):
    • (naive) 1x outbound network, 3x inbound network, 3x disk write
    • (smarter) 1x outbound, 2x inbound, 2x disk write

So the repair approach adds a 1x outbound network cost to the storage servers, and allows the repairer to use 1x less outbound bandwidth than the upload helper. Of course we'd want to upload a few extra servers to reduce the probability of a share being lost before we managed to turn around and repair the file back up to full strength.

reduced number of connections

I'm thinking about scaling. To get a petabyte of user data, we need something like 3k storage servers with a 1TB disk each. If you have 10k customer machines, that's 10k machines that need to be kept up to date on each of 3k servers. The number of connections is getting excessive (10k open SSL connections at once on each storage server), and if you want those connections to stay alive behind a NAT box you have to ping every few minutes, and 10k pings at 5min each is 33 pings per second.

More importantly I'm concerned that the peerlist-maintenance costs are going to get problematic, depending of course upon how frequently it's changing and how frequently the clients need to check for updates. If the client machines only talk to the upload helpers (and not the storage servers), that's an A+B vs A*B improvement: I'm thinking of a few hundred upload helpers, and each client only talks to one (or just a few) of them. That means you only have to update a few hundred machines about your new storage servers coming online, and the client only talks to one or a couple of machines.

If the number of storage servers gets large (millions), you could have a dedicated machine to do the peerid permutation, since that's going to benefit from keeping the whole list in memory.

Remember that bittorrent is a community per file: they may have hundreds or even thousands of people participating in a single file, but not 10k or 1M. There aren't sub-groups within the Tahoe node community: everybody needs to know about everybody. So I think the costs are going to be higher than what BT experiences.

• avoiding duplicate uploads of files which were uploaded by other users
• enabling resumable uploads
• improving simultaneous upload behavior

Aren't these features more or less independent of the off-loaded uploader?

I can't currently think of a way to make them independent. Specifically, I understand how to achieve them *with* the offloaded uploader, and I don't yet know how to achieve them without one.

The rough design we were sketching out on the board yesterday is that the client sends a (storage_index, crypttext-reader-object) pair to the helper. The helper asks the appropriate storage servers about the file, then it either returns the UEB hashes ("yes, the file has already been uploaded, here's the data you need to complete your URI if you want to be lazy and not check any of the hashes"), or it pulls crypttext from the reader object and saves it to a local file. When all the crypttext is there, it starts encoding and pushing. When it's done, it returns the UEB hashes. (note that the same helper could be used on the download side, to exchange a storage_index for a crypttext-reader, which wouldn't reduce bandwidth anywhere but would cut down on the number of connections and on the number of messages that have to traverse a high-latency client-side connection, and could enable some inter-client caching of a popular file).

In this scheme, uploads are resumable (meaning that the client might shut down or lose their network connection but won't have to re-transmit all of the previously-sent data) if they can connect to the same upload helper and if that helper still has their partial data. I assume that the upload helpers are on well-hosted machines in colo. When we have multiple upload helpers, they need to coordinate so that clients get routed back to the same one that still has the partial crypttext on disk. If the upload helper gets two clients who are uploading the same thing, it can grab some crypttext from one and some from the other (with appropriate design choices about how vulnerable you are to corrupted data, of course). The duplicate-upload avoidance could be done by a client node directly asking storage servers just as easily, the only advantage of this scheme is 1) fewer connections and 2) the helpers could conceivably cache the results for a while.

More importantly, I don't see how to cleanly achieve the resumable 1x uploads without an intermediate upload-helper. When I brainstorm about how this might work, it seems like we'd have to implement it in the HTTP layer (1x upload either requires sending plaintext over HTTP, or your upload-then-repair scheme, or uploading crypttext to a helper).. some alternative to PUT that can be resumed. We'd need to define an upload handle (the hash of the file, perhaps, or a random token which the client stashes somewhere), then a series of HTTP-but-not-RESTful operations to get file data to the tahoe node at the other end, then a POST operation to start the upload, and some more operations to find out what URI was produced. The combinations smells a lot like the XMLRPC-based "webfront" api we have on the MV codebase, and I'd really prefer to stick to the RESTful design because it's so straightforward and easy to understand.

To avoid duplicate uploads, we need to start with a file, reduce it down to a hash of some sort, ask a question using that hash, then construct a full URI. Either we have a server/oracle somewhere which will take the hash of a file and return the full URI of it (which makes me nervous, as well as being a server component that I don't want to build or require), or the client needs to do the same encrypt/encode (but *not* upload, it just discards the shares) as a normal tahoe node (so it can get the storage_index and the UEB hashes, unless it wants to be lazy and trust in the UEB hashes from the other end, but it still needs the code to hash the file into the encryption key). And, if the client has to have most of the tahoe code anyways, then why limit ourselves to HTTP for the thing-which-can-be-interrupted when we can use Foolscap and get better properties (and not mangle REST in the process).

So, I think a useful tool here would be some concrete proposals of how to achieve these goals without the offloaded-uploader approach. I haven't been able to get my brain around them.. if you've got a design in mind, that would help me a lot. I suspect one component of such a design would be a CHK storage server protocol which allows a client to fill the slot partway, then go away, then come back and fill the rest of the slot. I don't yet know how to do this securely.. maybe some kind of short-term shared secret that references the partial slot, and allows mutation (i.e. adding more data) until it is full, then the append-token goes away.

[warner]

Our current plans call for this to be implemented ASAP. The desireable properties of a grid in which most clients use this helper node are:

• 1x upload speed
• minimal connections: storage servers only know about helper nodes. Client machines only know about their one helper node. This also removes the overhead necessary to keep clients aware of all the storage servers (periodic peerlist updates).
• all plaintext and authorities remain on the client machines. Specifically, the design of the upload-helper does not require sharing any plaintext authority with external servers (their might be other reasons the client has chosen to share their authorities with an external server.. use of a plain web browser against a public web server node, for example)
• resume interrupted uploaders (partially-uploaded crypttext is kept on disk, keyed by storage index)
• avoid duplicate uploads (this would also be accomplished by having a local node which talks directly to the storage servers)

The overall design will be:

• if a tahoe node discovers a "encoder.furl" (or helper.furl, or some other yet-to-be-decided name) file in its base directory, then it will use the helper for uploads instead of talking to storage servers directly. It will also use it for downloads (to minimize connections, not to reduce bandwidth).
  • optional: maybe use the helper for uploads but not downloads. Maybe use the helper for large uploads but not small ones. Maybe use the helper for CHK files but not mutable files/dirnodes.
• if the tahoe node does not have an introducer.furl, it will not even be able to connect to storage servers directly, forcing it to use the helper for all uploads and downloads
• on CHK upload, the client will make a first read pass over the file to be uploaded, to compute the encryption key and storage index. It then sends the storage index to the helper
  • the helper might respond with "the file has already been uploaded, here is the UEB hash block", in which case the client can either:
    • (paranoid) perform a second read pass over the file, hashing and encrypting and encoding, creating all shares but discarding them rather than pushing them. From this pass it builds all the same hashes that a real upload would do. It compares these hashes against the UEB returned by the helper. If they match, it decides the file is safely present in the grid and uses the UEB hash plus the locally-generated encryption key to build the read-cap. If they do not match, it concludes that something is corrupted, and we need some sort of recovery mechanism (perhaps do a non-convergent full upload with a randomly-generated encryption key).
    • (trusting) accept the UEB hash block, build the write-cap, done
  • the helper might respond with "the file is not present in the grid", in which case it returns an object to which an !RICryptextReader should be given. This reader will be asked for crypttext from specific ranges. The helper object will write this crypttext to disk.
    • the indirection allows for two things: a farm of helper machines can talk amongst themselves to connect the client to the specific helper that has some partial crypttext sitting on disk, and the direction of Foolscap introduction is compatible with the client being behind a NAT box. Otherwise we'd just have the client provide an !RICryptextReader along with the storage index, and introduce the specific helper node to it instead of the other way around.
  • the helper also needs to be given the plaintext hash tree, since it must be included in the share/UEB.
    • TODO: it would be nice to minimize the amount of information that is revealed about the plaintext. Zooko would like to remove the plaintext and crypttext hash trees from the share, which would accomplish this goal.
  • when the helper has all the crypttext, it will begin encoding and pushing. Progress reports can be returned during this time.
  • when encode+push is done, the helper returns the UEB data (specifically crypttext hash tree root and share hash tree root) that it created.

In addition, we need to update the storage server protocol to make the three states of a share be visible (instead of collapsing them into only two): non-existent, incomplete, and finished.

[warner]

as I read the milestones, this actually belongs in 0.8.0

[warner]

since this is the only thing we have on the table to provide #218 (resume interrupted uploads), and it is the main plan we have for achieving 1x uploads, I'm raising the priority.

Client-side code is mostly done and pushed. I'm working on the server-side code now.

[warner]

server-side code works, system test passes. Needs a lot of cleanup. Next step:

• the helper should write the data to disk rather than streaming it all in RAM, to enable resumption.
• unit tests to exercise interrupt/resume
• unit tests to exercise multiple simultaneous uploads
• split server-side upload into query-for-existing-file phase and actual upload phase
  • to provide duplicate-upload avoidance
• test to make sure that a file which is already uploaded does not require transfer of data
• changes to create-node code to allow creation of Helper nodes

[warner]

another update: the client side code works, and now the helper writes data to disk before encoding. The interrupt-and-resume system test passes.

Next steps:

• unit tests to exercise multiple simultaneous uploads
• implement duplicate-upload avoidance
• unit test for duplicate-upload avoidance
• create-node mode to create Helper nodes more easily

[warner]

duplicate-upload avoidance code is in place, with a unit test. A quick manual test works.

What's left:

• unit tests to exercise multiple simultaneous uploads
• create-node to create Helper nodes more easily (this will probably be implemented after the Introducer refactoring)
• stats on number of pending uploads, etc (after rob's stats-gatherer code goes in)

[warner]

since all the main functionality is in place, I'm reducing the priority of this one. We could ship today with no further improvements to this particular component.

[zooko]

Shall we close this ticket and open a new one for the bits that are left?

[warner]

um, sure. this ticket is getting stale anyways :) The new #333 has these remaining items.