Goals for new filecaps

This is a place to record desiderata for the next version of our mutable/immutable filecaps. Many of the design requirements are spread out across separate tickets: this page is here to consolidate them. We should not release a new filecap format without checking it against everything on this list.

There will be a related pair of new encoding designs. The NewImmutableEncodingDesign and NewMutableEncodingDesign pages will hold those design discussions.

Ticket #432 was the starting point: it contained a list of features.

make them real URIs

Kevin Reid points out that the Tahoe calls URIs are not actually URIs (in the established sense). To make them real, we need to:

• make them start with x-tahoe: or tahoe: (or lafs:), register tahoe: with IANA (#418) (#683)
• understand how URI/URL/URNs are built, decide about hierarchical segments vs non-hierarchical segments. What's magical about a leading double-slash? Do we need one?
• according to #683, a URI identifies a resource, but does not necessarily provide enough information to actually access it (i.e. if you have a URI and somebody pointed you at a file, you could confidently tell them whether or not it was the right file, but if you only have the URI, then you might not be able to find the file without additional information). If the cap has both identifying and location information, it's called a URL.
• Tahoe filecaps are meant to be URLs (they are intended to provide location information), but to really make that work, you also need to define which grid you're talking about. So far this has always been implicit, but that has caused us problems. #403 talks about making an explicit "gridid" and would provide a procedure to get from a gridid string to a set of storage servers. The existing tahoe codebase could use the introducer FURL as a gridid, if there were a good place to put it in the filecap (#683 touches on this).
  • from the point of view of a web browser, you also need a gateway service (the Tahoe client node with a webapi frontend). The tahoe URLs that we've been passing around so far always reference one of these, either by assuming that http://localhost:8123 is a suitable gateway or by explicitly referencing an external gateway like testgrid.allmydata.org (with deleterious effects on security and availability). I hope that our new filecaps are defined independently of a webapi gateway used to access them, and that we have a clear procedure for starting with a filecap and a gateway HTTP URL, and ending with the contents of the file.

make them shorter, prettier, and easier to use

• Short and not so ugly. This is important to enable cut-and-paste (see below), but also just because people are suspicious and averse to long and ugly URLs. See #217 for notes in which dozens of people have spontaneously complained about the current URLs. By contrast, tiny URLs such as tinyurl.com, bit.ly, etc. are ubiquitous nowadays; users have no problem with those -- see Twitter.
  • I (warner) am curious about where the suspicion comes from. Do long URLs make people think they're being attacked, some sort of browser buffer overrun thing? Or that they're being phished, with a URL that a human would evaluate differently than their browser? I agree that people (including me) don't like long URLs, but I've never pushed anyone to explain the "suspicion" aspect. One comment in #217 says "smells a bit spammy", and a later one says "Spooks me every time".
• Enable convenient cut-and-paste. If caps are too long they'll wrap in email. If they contain lots of word-breaking characters then you have to drag after you've double clicked (this is probably ok). If the word-broken sections are small and at the beginning or end then you have to be very precise about that drag. The best design would be a single short non-word-breaking string. The next best will be to have a large non-word-breaking string at the start and end, with smaller segments (if necessary) in the middle. Note that tahoe: is an easy target, but x-tahoe: is not (you'd have to double-click on the "x").
• Usable in a browser. Specifically, it should be easy to actually use a filecap that you get in email or IM, and many email/IM clients will look for http URLs and make them clickable. If tahoe filecaps start with http:, then they'll be made clickable. This is at odds with the IANA-friendly tahoe: prefix. Clients may make tahoe: URIs clickable too (I've seen them make other letters-then-colon strings clickable, even when the letters are not "http"), so perhaps a reasonable solution is to provide an OS-level URI handler for the tahoe: scheme, which could embed the filecap in an http URL and submit it to a webbrowser (i.e. when you click on tahoe:foo, a helper program is launched with tahoe:foo, and that in turn launches your web browser with http://localhost:8123/foo). (#52)

make them long enough to be secure

We want filecaps to be as short as possible, but no shorter. There are several lower bounds on the length:

• confidentiality: A large computing effort should not be able to obtain the plaintext of a tahoe file without knowing the readcap. We require reasonable margin against improvements in hardware speed and organizational efficiency/motivation of distributed efforts (e.g. could a million PS3 owners break a filecap?). This currently implies a 128 bit confidentiality field.
• integrity: a large computing effort should not be able to produce shares which will be accepted by the readcap holder but which do not result in the same file as created the original uploader (and retrieved by other downloaders). We desire all three of the standard hash properties (collision resistance, first-pre-image resistance, second-pre-image resistance) to also apply to tahoe immutable files and their filecaps. This currently implies a 128bit (or 256bit?) integrity field.
  • variable-length integrity field (#102, comment 16+17), allowing users to decide between short caps and strong integrity guarantees
• storage collision resistance (#753): a Tahoe grid should be able to store trillions of files and still have a vanishingly small chance of two files using the same storage-index (and thus confusing each other's shares). The storage-index is generally compressed out of the filecap, by deriving it with various hashing stages on the other filecap parameters. The shortest value in this derivation chain must be at least 128bits long, and preferably about 192bits long.

other features

• Self-identifying. It should be visually clear what sort of filecap the string represents: read-write or read-only, mutable-or-immutable, file-or-directory. This is especially important when sharing tahoe objects over out-of-band channels like IM and email: it should be easy for the user to tell whether they're giving away readonly access or read-write access. We've considered prefixes like DWM.. for "Directory Writeable Mutable" and FRI.. for "File Readonly Immutable" (#102 comment 12). If these are jammed against the (base62) crypto bits it may be difficult to tell where the prefix ends and the crypto bits begin, especially because the crypto bits will be using the same character set (FRIDWM...). It might be a good idea to separate the type prefix from the cryptobits: FRI-cryptobits or FRI/cryptobits.
  • in addition, tahoe URIs should be distinguishable from local filenames by a CLI tool, so that tahoe cp $CAP local/foo.txt is unambiguous. (unfortunately, the current practice of using "tahoe:" as a default alias name collides with this badly, but perhaps if the new URIs include the double-slash, this won't be a problem: tahoe cp tahoe://CAP local/foo.txt copies from a specific URI, while tahoe cp tahoe:blah local/foo.txt copies from a child of the "tahoe:" alias).
  • I'd like to make it easy to layer uses on top of one another: since directories are just a specific way of interpreting the contents of a (mutable) file, let's make the directory cap be closely related to the underlying filecap. For example, if we end up using tahoe://MR/cryptobits to describe a read-only mutable file referenced by "cryptobits", then we could use tahoe://D/MR/cryptobits for the directory that uses it as a backing store. The rule would be that tahoe://D/$A would be handled by fetching tahoe://$A and then interpreting its contents as a directory structure. Then reading immutable-dirnodes (#607) would be trivial. Another way to think about this is that if our filecaps were verbose s-expressions, these caps could be expressed as "(readonly (mutable cryptobits))" and "(directory (readonly (mutable cryptobits)))".
• provide for verifycaps, repaircaps, and traversalcaps (#308, #217). Repaircaps in particular may require a grant of storage authority, which might entail a cap format that can accept arbitrary extra non-hierarchical fields. Appendcaps or "drop-box" writecaps might fall into this same space. But remember that URIs should identify objects, not the action that you want to do on it: a webapi scheme may use a POST/PUT/DELETE method, or append a t=json adverb, or alternatively encode the verb/adverb into the HTTP url (think GET .../filecap/json or PUT unlinked/ciphertext), but these are independent of the underlying filecap. Exotic cap types do not need to be short.
• provide ciphertext access. Reading from a verifycap should give you ciphertext. It should be possible to upload ciphertext directly.
• provide for a grid-identifier, possibly on the MSB end, e.g. tahoe://grid1234/IR/cryptobits. Perhaps let some contexts define a "default grid id", such that tahoe://IR/cryptobits is expanded to mean tahoe://grid1234/IR/cryptobits. Something like tahoe://grid1234/D/MR/cryptobits should reference tahoe://grid1234/MR/cryptobits. (#403)
• permit multiple encodings of the same file (same k, different N) to use each other's shares (#678)
• be derived from a hash of the plaintext to detect bugs in the decryption or erasure-decoding (#453) and possibly also to be able to easily give users a hash of the plaintext if they want it (#280)