LAFS 301 Moved Permanently

[zooko]

Make it possible to mark a cap as moved-permanently, including possibly to a cap of a newer version.

Described in pipermail/tahoe-dev/2010-February/003910.html as follows:

Dear people of tahoe-dev:

I've been hosting my blog on Tahoe-LAFS since July, 2008. That's 18
months now! Amazing. There are now a lot of copies out there of the
URL to my blog, which is:
http://testgrid.allmydata.org:3567/uri/URI:DIR2-RO:j74uhg25nwdpjpacl6rkat2yhm:kav7ijeft5h7r7rxdp5bgtlt3viv32yabqajkrdykozia5544jqa/wiki.html
. (The URL to my blog includes, of course, the read-cap to my blog.)

I was thinking recently about what will happen we deploy New Cap
Design and I want to upgrade my blog to take advantage of the new
style caps. When I do so the read-cap will have to change. I would
want to preserve the unforgeability property -- the guarantee that if
you have the read-cap to my blog you can't be tricked into accepting a
forged document that was generated without the write-cap to my blog.

Of course, I could just add a new blog entry in my old blog saying "My
blog has now moved to $HERE, please update your bookmarks
accordingly.", with the new read-cap. This would preserve the
unforgeability property, but it can take a long time for everyone in
the world to update their bookmarks, and the longer it takes the
longer those people would be vulnerable to forgeries (by an attacker
who succeeds at cracking the RSA digital signatures that provide the
unforgeabilithy of my old blog). Also, asking them to do this is a
hassle, so I would not do it unless the old cap style had become
*really* old and troublesome -- I would keep using the old cap for a
long time just to avoid this "manually asking people to change their
links".

Also, this text "My blog has now moved to $HERE" is not standardized
and machine-readable, so you can't write a script that will reliably
detect this sort of note and update your bookmarks for you.

Then I had an idea: secure HTTP 301 Moved Permanently!

We could extend the Tahoe-LAFS cap format so that when you ask for the
current version, there is a special symbol along with a new read-cap
(out-of-band of the file contents so that there is no ambiguity about
file contents versus the 301 Moved Permanently symbol). This special
symbol means that the client should download the new read-cap,
remember for future reference that any attempt to load the old
read-cap should instead use the new read-cap, and then (if possible)
fetch the contents of the file using the new readcap.

Note that this dovetails very nicely with my previous proposal for
write-cap revocation (#954). If you are going to issue a secure LAFS
301 Moved Permanently, then you also want to petrify that file so that
nobody can later change it.

Note that the part about the client "remembering for future reference"
suggests the existence of a client-side database which records the
fact that a specific URL has been 301'ed and optimizes out the step of
looking up the old location next time. This might fit in well with the
backupdb, which already likes to remember a few key facts about files
and directories to optimize backups. It might also be a good idea to
update parent directories which point to that file or directory with
the 301 information (#956).

This is a forward-compatibility issue because, as in the case of my
blog, the sooner we can rely on this feature the easier we can upgrade
to future versions of caps. I've created #958 to track this issue.

Regards,

Zooko

• #954# revoke write authority
• #955# use client-side storage to defend against rollback attack
• #956# embed security metadata in parent directory
• #957# embed security metadata in URL
• #958# LAFS 301 Moved Permanently

[davidsarah]

As a backward-compatibility hack to allow users of older webapi servers to be able to access the target of the redirection, we can store it as a directory, with one of its children specifying that it is a redirection link (for example, by using a "redirection: true" entry in the "tahoe" metadata).

This would cause newer servers to return a 301 Moved Permanently response (maintaining any subsequent URL path elements). Users of older servers would see this just as a child of the directory, and could follow it manually. The other children could be used to maintain the validity of existing links -- for instance an URL ending in /wiki.html would end up pointing to a petrified version of that file.

This wouldn't work if the original URI is to an immutable file or directory. However, if it is to a mutable directory (which is probably the common case, and is true for Zooko's blog URL for example), then it would be possible to direct it to a new mutable or immutable directory.

[davidsarah]

Replying to davidsarah:

for example, by using a "redirection: true" entry in the "tahoe" metadata

Note that this implementation would require a new webapi operation to set this field of the metadata. Whether we use a "tahoe" subkey is independent of the rest of the idea; adding a new operation for redirection obviously adds some complexity, but it may be desirable to ensure that this can't occur accidentally when using existing APIs.

[davidsarah]

Proposed CLI command:

tahoe permanent-redirect <olddir> <newdir>

Redirects <olddir> permanently to <newdir>. <olddir> must refer
to a mutable directory; <newdir> may refer to either a mutable
or immutable directory.

This operation can be used to revoke write access to <olddir>.
The security of this revocation is dependent on the cooperation
of storage servers -- an attacker would need to have the write-cap
for <olddir> and control any K servers, in order to be able to
write to the directory as seen by a client using that value of K.

Backward-compatibility notes:

All storage servers must be running Tahoe-LAFS v1.7 or later,
otherwise this command will fail. Storage clients running a
version before v1.7 will not automatically follow the redirection;
they will see a shallow-frozen copy of <newdir> with an extra
child called "redirect" (any existing "redirect" child will be
replaced). Therefore, you should make sure that the current
children of <newdir> reflect what you want these clients to see.
For security, any subdirectories of <newdir> should preferably
be immutable at the time you use "tahoe permanent-redirect".

Clients running Tahoe-LAFS v1.7 or later will follow the
redirection automatically and will not see any extra link.
There is a limit on the length of a chain of redirections
clients will follow before giving an error.

[davidsarah]

Note that for HTTP clients, it's normally the client's responsibility to follow redirects. However the CLI commands (which are webapi clients) don't currently do so. Should we have the webapi server follow redirects in order to simplify webapi clients? It would have to follow redirects anyway in order to implement non-HTTP frontends (SFTP, FTP and SMB).

[davidsarah]

Replying to davidsarah:

an attacker would need to have the write-cap for <olddir> and control any K servers, in order to be able to write to the directory as seen by a client using that value of K.

Ah, no they wouldn't. A single storage server might have K distinct shares. To make the above statement true, we'd have to change download so that it always uses K servers (at some cost in availability).

[davidsarah]

Replying to davidsarah:

Should we have the webapi server follow redirects in order to simplify webapi clients?

No, we can't, because that would defeat the point of using a redirect (as a machine-readable way to signal that the URL has been updated, and in the case of browsers, that the new URL should be shown in the address bar).

It [the webapi server] would have to follow redirects anyway in order to implement non-HTTP frontends (SFTP, FTP and SMB).

I think we'll just have to do it in both the webapi server and clients (since these other protocols have no way to express a redirect).

[davidsarah]

Replying to davidsarah:

Note that for HTTP clients, it's normally the client's responsibility to follow redirects. However the CLI commands (which are webapi clients) don't currently do so.

They don't because source:src/allmydata/scripts/common_http.py​ uses httplib directly. But this is easy to fix, for example by using urllib2 as described ​here.

So we should open another ticket to make the CLI commands follow redirects (if we're correct that they don't).

[davidsarah]

The ticket to make the CLI commands follow redirects is #965.

[zooko]

Replying to davidsarah:

All storage servers must be running Tahoe-LAFS v1.7 or later, otherwise this command will fail.

Maybe this backwards-incompatibility won't be necessary! I think there might be an extensible structure, the contents of which the storage servers will pass on to downloaders without needing to understand the contents themselves. In particular, I think maybe the UEB (someday to be renamed CEB when "URI" gets renamed "Capability" everywhere) can hold the new 301 redirection marker and the new cap that this cap has been redirected to.

[davidsarah]

Replying to zooko:

Replying to davidsarah:

All storage servers must be running Tahoe-LAFS v1.7 or later, otherwise this command will fail.

Maybe this backwards-incompatibility won't be necessary! I think there might be an extensible structure, the contents of which the storage servers will pass on to downloaders without needing to understand the contents themselves.

How would older servers know to refuse to update a share? Temporary redirect wouldn't require upgrading servers, but we can't see how to avoid that for permanent redirect (if it is to be usable for revocation).

In particular, I think maybe the UEB (someday to be renamed CEB when "URI" gets renamed "Capability" everywhere) can hold the new 301 redirection marker and the new cap that this cap has been redirected to.

Yes, that's a viable alternative design to putting the marker in the metadata, but it doesn't seem to solve the problem of older servers still accepting share updates.

[zooko]

We're out of time to do this for v1.7.0. I hope to do it for v1.8.0.

[zooko]

I'm interested in prioritizing this ticket because it is (IMHO) a major forward-compatibility feature. But, I haven't really looked at all of the other competing priorities for v1.8.0 and decided that I'm really going to spend my time on this one, so I'm putting it in the "soon" Milestone for now.

[zooko]

Hm, would it be okay to allow people to set an HTTP 301 to a different cap of a different type, such as a read-write cap instead of a read-only cap or a read-only cap instead of a read-write cap?

Our tradition of transitive attenuation of authority suggests that we should forbid this, which means that a client which is following an HTTP 301 redirect should remember whatever the attenuation of the original cap was (i.e. if it was read-only or ??? if it was a verify-only cap) and refuse to use the new cap with authority outside of that.

[davidsarah]

Replying to zooko:

Hm, would it be okay to allow people to set an HTTP 301 to a different cap of a different type, such as a read-write cap instead of a read-only cap or a read-only cap instead of a read-write cap?

Allowing redirection of a read-write cap to a read-only cap seems harmless and useful.

(More precisely: as long as we have API slots that can accept either a read-write cap or a read-only cap, we should accept rw-redirecting-to-ro in those slots. There's an argument for having more strictly typed APIs, for which read operations reject write caps, in order to catch inadvertent grants of excess authority; but that's orthogonal to redirection.)

Our tradition of transitive attenuation of authority suggests that we should forbid this, which means that a client which is following an HTTP 301 redirect should remember whatever the attenuation of the original cap was (i.e. if it was read-only or ??? if it was a verify-only cap) and refuse to use the new cap with authority outside of that.

+1 (for forbidding redirection of read-only caps to read-write caps).

This should also apply to verify-only caps in the contexts in which they can be used, e.g. ?t=info, checker APIs, etc.