[tahoe-dev] design heads-up: moving lease data out of sharefiles

[Brian Warner]

While going over my Accounting work[1] this morning, I had an idea about
simplifying the backend storage share-file format. I'd like to remove
the lease information from the share files themselves, and use a
separate per-server sqlite database (the "LeaseDB") to hold all lease
data. I wanted to mention it right away since it interacts with other
folk's design work, in particular Least Authority Enterprises (probably
for the better: I think it'll make their job simpler).

So far, we've stored all information about leases in the same file as
the share data. The general idea is that the share file is canonical,
and we have a bunch of Crawlers whose job it is to
update/refresh/maintain secondary data structures for faster access. We
did this so that we could manually move shares from one server to
another by simply copying the backend files with 'scp' or the like, and
all the metadata would travel along with them.

But having the lease data in those files is a hassle: it's variable
length, which means shares for immutable files will change size over
time. It requires storing per-server renew/cancel secrets for each
share, which both hampers actual migration (the secrets end up being
wrong) and means that part of the share file should be kept secret from
readers (which was the cause of the security bug that prompted 1.8.3).
And Accounting needs that lease data to be in a place where it can be
summarized quickly (to answer requests like "find all expired leases",
"find all shares that I hold leases on", "how much space am I using", or
"Bob shut down his account, cancel all his leases right away"), for
which a real database with a proper index works a lot better than a
terabyte of share data with tiny bits of lease scattered throughout.

So my proposal is this:

 * make the LeaseDB be the canonical source of lease information
 * stop updating, ignore all lease info in the share files
 * build an AccountingCrawler with a schema like this:
   CREATE TABLE buckets -- I think David-Sarah proposed "sharesets" here
   (
    `storage_index` VARCHAR(26),
    `id` INTEGER PRIMARY KEY AUTOINCREMENT
   );
   CREATE TABLE leases
   (
    `bucket_id` INTEGER,
    `account_id` INTEGER,
    `expiration_time` INTEGER
   );
 * when the crawler sees a share that isn't in the "bucket" table, add
   it, and add a special "migration lease" that lasts for a month or
   two. This handles shares that were copied in manually, and also the
   transition period when the server is first upgraded to this code.
 * if the crawler sees an entry in the bucket table that has no actual
   shares on disk, delete the entry. This handles shares that were
   manually deleted.

With that, Accounting can work by adding entries to the "leases" table.
Periodic expiration will query the table to find all leases with
expiration_time in the past, make a list of their bucket_ids, delete
those leases, then walk the list of potential victims to see which ones
have no more leases left, and delete the shares (and "buckets" entry) if
so configured.

This will allow basic manual share migration to work well enough: the
requirement is that the client does a deep-add-lease within a month or
two to reclaim their shares (which GC requires anyways). It might
increase the server's storage burden slightly, if the migrated shares
were about to expire on their own (they'll last a month or two extra),
but I think that's minor given that manual migration is not likely to
happen frequently.

And I think we can build better migration tools anyways: imagine a
"tahoe storage export SHARE-CRITERIA.." command, which streams data to
stdout, to be caught by a corresponding "tahoe storage import" command.
The data could include both the share data and the lease information, in
a form that is meaningful to the remote end (translating local
account_ids, for example, or having the old *server* claim a temporary
lease on the shares, attributing the storage burden to them until the
real owner takes over). Or a form which copies the export data to a
removable disk, to then be manually transported to the new server.

I want this because the AccountingCrawler that I've half-implemented so
far is worryingly complex, as it's trying to keep a database table in
sync with the sharefiles' embedded lease information. And because
keeping the leases inside the shares has been a PITA anyways :).

Thoughts?
 -Brian


[1]: https://github.com/warner/tahoe-lafs/tree/accounting