[tahoe-dev] protocol versioning (v2, as it were)

[Brian Warner]

Zooko and just got off another phone call, and we think we've converged
on a protocol versioning scheme. This one is specific to our current
Foolscap-based connections, but will probably have uses beyond that.

Each primary Foolscap-reachable object (IntroducerService, StorageServer,
Helper) will have a remote_get_version() method, which takes no arguments and
returns a dictionary (with string keys and arbitrary values).

Each key will refer to a feature or set of features. The actual string keys
of this dictionary will be URLs, each of which points to a human-readable web
page that contains a brief description of the feature and how it should be
interpreted. This is really intended for humans, so they can learn more about
the reason that their client wasn't able to talk to a server.

Clients will use this dictionary to decide how to interact with the server in
question. When the client first connects to the server, it will call
get_version() and stash the response dictionary in a connection-specific
place (i.e. the response is cached for the life of the TCP connection). If
the get_version() method is not implemented (i.e. callRemote("get_version")
results in either NameError or foolscap.Violation), the client will
substitute a default dictionary, which contains a feature set that is
equivalent to some reasonable description of what older servers provided.

The response dictionary will always contain an "application-version" key,
whose value is an ascii string that will more-or-less indicate which source
code is providing the service. A client with arbitrarily-copious information
and effort could use the application-version to figure out behavior
differences that were not anticipated by the protocol-version definers.
Anticipated changes, of course, would be expressed by the other keys.

Here's how we're expecting to use this scheme for the next few releases:

== IntroducerService ==

The current introducer code uses unsigned announcements. I have code in a
sandbox which uses signed announcements (such that clients will only use
servers which have been "blessed" by a client-specified public key), that is
waiting for the next pycryptopp release before it goes into trunk. Some day,
all announcements will be signed and we'll remove the non-signing code.

v1.2.0 and earlier did not offer get_version(). The v1.3.0
IntroducerService.get_version() will return:

  { "http://allmydata.org/tahoe/protocols/introducer/v1" : {},
    "application-version": "allmydata-tahoe-1.3.0" }

  "v1" means: remote_publish() and remote_subscribe(), non-signed messages.

When we get the #466 signed-introducer code in place, the new introducer
will return:

  { "http://allmydata.org/tahoe/protocols/introducer/v1" : {},
    "http://allmydata.org/tahoe/protocols/introducer/v2" : {},
    "application-version": "allmydata-tahoe-???" }

  "v2" means: remote_publish_v2(), remote_subscribe_v2(), signed messages

The v1.2.0 (and earlier) client will not call get_version(), instead it will
just blindly use the old remote_publish() and remote_subscribe() APIs. This
will work normally until some future day when we remove the old interfaces
from the server code, at which point these by-then-ancient clients will get a
weird exception at startup.

The v1.3.0 client will call get_version() with the v1.3.0 IntroducerService
response as a default. If it sees the introducer/v1 key in the response
dictionary, it will use the introducer as normal. If it does not, it will
raise an exception that says "I need the http://...v1 service, and this
server does not offer it. The server's version data is:...". This should give
the user enough information to learn about the various introducer protocols,
and discover that their client needs to be upgraded. The introducer
connection is critical to Tahoe functionality, so this exception should
terminate the program.

The (e.g.) v1.4.0 clients that know how to speak v2 will call get_version(),
if they see v2 in the advertised list, they'll use it, else if they see v1
they'll use it, else they'll raise an exception saying they need v2 or v1 but
they got ... .

== StorageServer ==

The current storage server code offers methods like allocate_buckets() and
(mutable) writev(). It also has an immutable-share-size limitation of about
4GiB. The next planned change to the storage server will be to remove the
4GiB limit (actually, we'll raise it to about 18EiB). Then, when accounting
is implemented, we'll change some part of the interface to add
storage-authority arguments to allocate_buckets() and writev(). At some
point, we'd like to change the storage API to allow a more streaming
interface: this might involve setting part of the storage index after the
data has been written. We might also like to switch to an API with fewer
round trips, which would remove allocate_buckets() in favor of some sort of
immutable_write() call.

So, v1.2.0 and earlier did not offer get_version(). The v1.3.0
StorageServer.get_version() will return:

  { "http://allmydata.org/tahoe/protocols/storage/v1" :
     { "maximum-immutable-share-size": 2**32 },
    "application-version": "allmydata-tahoe-1.3.0" }

The "maximum-immutable-share-size" indicates that clients should not attempt
to use allocate_buckets() to create shares which are larger than this. (in
fact, v1.3.0 will throw an exception when this is attempted, and v1.2.0 and
earlier will silently corrupt such large shares; the actual change occurred
somewhere in between the two releases).

When we fix the 4GiB limit (by defining a new backend share format, with
8-byte size fields), the new announcement will look like:

  { "http://allmydata.org/tahoe/protocols/storage/v1" :
     { "maximum-immutable-share-size": 2**64 },
    "application-version": "allmydata-tahoe-???" }

If we implement accounting with a new set of remote calls (as opposed to a
separate "storage-login" service, to which you'd pass storage authority and
then get back a storage facet), then we'd define a new "v2" feature set (with
an allocate_buckets_with_authority method), and the announcement would look
like:

  { "http://allmydata.org/tahoe/protocols/storage/v1" :
     { "maximum-immutable-share-size": 2**64 },
    "http://allmydata.org/tahoe/protocols/storage/v2" :
     { "maximum-immutable-share-size": 2**64 },
    "application-version": "allmydata-tahoe-???" }

If we subsequently add a more streaming interface (perhaps with methods like
write(upload_handle, data, finish, storage_index) that could be called
several times before finish=True), we'd define a "v3" feature set, and the
announcement would look like:

  { "http://allmydata.org/tahoe/protocols/storage/v1" :
     { "maximum-immutable-share-size": 2**64 },
    "http://allmydata.org/tahoe/protocols/storage/v2" :
     { "maximum-immutable-share-size": 2**64 },
    "http://allmydata.org/tahoe/protocols/storage/v3" :
     { "maximum-immutable-share-size": 2**64 },
    "application-version": "allmydata-tahoe-???" }

Then, some day, if we drop support for the older protocols, the announcement
would become:

  { "http://allmydata.org/tahoe/protocols/storage/v3" :
     { "maximum-immutable-share-size": 2**64 },
    "application-version": "allmydata-tahoe-???" }

Clients will call get_version(), and upon Violation or NameError will assume
the remote end said:

  { "http://allmydata.org/tahoe/protocols/storage/v1" :
     { "maximum-immutable-share-size": 2**32 },
    "application-version": "allmydata-tahoe-before-1.3.0" }

So v1.2.0 clients (which do not call get_version()) will blindly upload their
shares regardless of size (and might get an error, or might get corruption).
v1.3.0 clients will skip servers who don't advertise support for large enough
shares (which may cause uploads of large files to fail). v1.4.0 clients that
want to use storage authority will use the v2 protocol on servers which
support it, but will fall back to v1 on servers which don't.



So, anyways, that's the current plan. Thoughts and ideas are welcome!

cheers,
 -Brian