[tahoe-dev] Accounting 11.10 edition: working demo!

[Brian Warner]

I was able to get basic Accounting working last night. This actually hit
90% of the goals of my previous "Accounting, 2010 edition" message from
last December[1]. If you'd like to try it for yourself, do this:

 1: install http://github.com/warner/python-ed25519
 2: git clone -b accounting https://github.com/warner/tahoe-lafs.git
 3: cd tahoe-lafs
 4: start a grid with this version (you want a new Introducer, client,
    and servers)

You'll see two big tables on the new Welcome page (screenshots
attached). The old server table has some new columns added (how much
we're using on their server, Account Status, and Server Message). The
new client table has columns for current connection status, how long
they've been connected or disconnected, when they were first created,
and how much they're using on this server.

(remember to do this on a throwaway grid: all sorts of incompatible
changes will happen in the future)

== Simplifications ==

Relative to my last status update, I've made the following
simplifications:

* store all leases in a sqlite database. The lease fields in the storage
  shares are ignored and not updated. Related code was deleted.
* remove all renew-secrets and cancel-secrets
* allow at most one lease per Account. You can create/renew it, but you
  can't cancel it.
* remove expiration for now, it will be added back in before we're done

== #466 Signed-Dictionary Announcements ==

The Accounting branch builds on top of my #466 branch, which in turn
builds on some more #1363 server-id cleanup work. Servers publish signed
annoucement dictionaries to the Introducer. The Introducer relays the
signed message to all subscribers. Subscribers receive the messages,
verify the signature, and deliver the dictionary to their local
callbacks (generally just the StorageFarmBroker, which extracts the
Storage Server details and establishes a connection).

Backwards compatibility works pretty well. When a server detects an
older Introducer, it falls back to publishing unsigned tuples. When a
new Introducer receives unsigned tuples from an old server, it
translates them into (unsigned) dictionaries for newer subscribers, and
relays the unsigned tuples to older subscribers. Old subscribers connect
to the Introducer with a method that reveals their unawareness, and the
Introducer converts signed dictionaries into unsigned tuples for them.
If you have new code, you get the new behavior, if there is any old code
in the way, you get the old behavior.

I've got two variations: one with an embedded copy of my python-ecdsa
code, and another that uses an externally-provided python-ed25519. I
like the latter much better: faster, smaller keys, safer (doesn't depend
upon entropy during signing), deterministic. The Accounting branch on
github is currently based on the ed25519 variant.

== Keys ==

Each server creates an Ed25519 signing key, and uses it to sign its
Introducer Announcements (this is #466). Servers that are new enough to
understand Accounting advertise a second FURL named "accountant-FURL"
which points to an Account-Facet-creating service. For now, they also
advertise an "anonymous-storage-FURL" for older clients.

Each client creates a Ed25519 signing key. Knowledge of this key
represents authority to consume space under a specific "Account".
Clients connect to each storage server's "accountant-FURL" (if
advertised), then sends a signed message requesting that a reference to
client's Account object be delivered to a return-address FURL. Once they
receive this Account reference, they use it just like the old
RIStorageServer (but it has the account information curried in, and
applied to each operation). Older clients, or newer clients faced with
older storage servers, will connect to an "anonymous-storage-FURL",
which is an RIStorageServer hardwired to a special "anonymous" account.
Once everyone is using Accounting, server admins can disable the
anonymous account.

#466 reduces the use of Foolscap TubIDs significantly (the notion of
"peerid" or "serverid" is broken up into smaller separate uses).
Write-enabler secrets are still keyed by Foolscap TubIDs, since that's
what really matters to protect those secrets in flight. Server-selection
permutation is keyed by a server-chosen seed (in #466 the announcement
has a "server-permutation-seed" field, which is populated with the TubID
if the server has any shares, otherwise is populated with the server's
verifying key).


== Accounting Model ==

>From a client's point of view, each server (identified by the key that
signed its announcement, or by a TubID for unsigned announcements) gives
the client an "Account", which has several properties:

 * current usage (in bytes)
 * Status: 3 booleans: is the server willing to accept new shares?
   willing to provide access to existing shares? willing to retain (not
   delete) existing shares? Clients in good standing will get
   (True,True,True), clients who have used up their quota will get FTT,
   clients who've really ticked someone off will get FFT ("we're holding
   your data hostage") or even FFF ("we deleted all your data")
 * Server Message: still kinda fuzzy, the general idea is to explain
   what payment/get-more-space options you have

The client gets access to an Account object, which acts like an
RIStorageServer (so provides share upload/download and mutable-share
readv/writev) but adds methods to inspect these properties.

>From the server's point of view, the client connects to their
"Accountant Window" and sends a signed message asking for an Account
object. The message includes a "return address" FURL: the server
connects to that FURL and gives it a reference to their per-client
Account object (this ensures that only someone who knows the client's
private key can get access to the client's Account object). The Account
acts like a normal RIStorageServer, but every time they consume space, a
lease is added, marked with their account.

== LeaseDB ==

NODEDIR/storage/leasedb.sqlite contains all information about client
Accounts and share leases (see schema here[2]). Conceptually, there is a
grid with Accounts on one axis (specifically a base32 verifying-key),
and "share-IDs" (storage-index and shnum) on the other axis. Each
intersection is either empty (no lease) or contains a lease with an
expiration time. Each share-ID row also knows the share's current size.

An account's current usage is obtained by summing the share sizes of all
share-IDs for which that account holds a lease. Expiration will happen
by deleting all leases with old expiration times and then find+delete
all shares with no leases (I haven't written this code yet).

There is also an Account table, which remembers things like when the
account was created, when (and from where) they last connected, and the
nickname they offered. We can also add local notes to it, like why we
decided to grant storage access to them, a petname, and information
about reciprocal storage offers: we'll give them space if they'll give
us space, connecting outward-looking storage_client.IServer instances
with inward-looking storage.accounting.Account instances (still very
much in the air).

All decisions about leases and expiration can be controlled directly
through the database, so in the future we could run expiration or
account-usage calculations from a different thread or process, reducing
the performance hit somewhat. In particular, measuring usage of all
accounts involves some big DB queries, so it might be best performed
from a different process. The share-crawler will probably be responsible
for performing actual share deletion.

== Migration ==

A share-crawler notices shares on disk that are not present in the
database, and gives them a two-month "starter lease". This keeps those
shares alive until their clients do a deep-add-lease and claim them
properly. This takes care of startup (what happens just after you've
upgraded the server to a code version that knows about Accounting) as
well as manual transplantation of shares from other servers. These
leases are measured under the "starter" pseudo-account. In the long run
(when real Accounts establish leases on live shares, and no manual
share-transplantation is going on), this account should have zero usage.

Old clients who don't know about Accounting, or who hear about the
server through an older (non-#466-aware) Introducer, will connect to the
"anonymous" pseudo-account. All their leases will be measured under this
account. In the long run (when all clients are using real Accounts),
this account should have zero usage, and then can be disabled (so old
non-Accounting-aware clients won't be able to connect at all).

== Necessary Improvements ==

I think the "all clients" and "all servers" pages should be split off
the front Welcome page, leaving just a summary ("connected to 6 out of
10 known servers", "4 out of 8 known clients connected") and links to
the sub-pages.

We're going to need some sort of control panel. The present code gives
free full-access accounts to everybody and merely measures how much
space they're using. Really, we want server admins to be able to set
quotas, disable accounts, etc. So we either need config-file-based
NODEDIR/tahoe.cfg -style control knobs, or a private+secure web page
with buttons (feels best from a UI POV), or some other kind of UI
(wxWidgets? pygtk? curses?).

Storage authority is currently ambient within a given gateway: all files
uploaded from that gateway will fall under the same Account. At the very
least we'll want to make it easy to turn off space-consuming operations
(the "read-only gateway" switch that several folks have asked for).

== Unnecessary Improvements ==

We can afford to defer some improvements for a while:

* multiple leases per account. I'm thinking of having a non-secret
  "lease id" on each lease, a free-form string the client can use for
  their own purposes. You renew specific lease-ids. This would make it
  safer to bring back cancel-lease, by reducing the coordination
  requirements for things like:
  * multiple clients that share the same account
  * multiple rootcaps within a single client
* delegation: the old renew-secret and cancel-secrets were carefully
  derived to allow you to hand all your lease-renewal authority to
  someone else with just a single string, which you might use to keep
  your shares alive while going on vacation. We could add a
  Account-protected operation to add one or more verifying keys to each
  lease, and then you could hand out the corresponding signing keys to
  delegate renewal authority. Or we could use certchains (signed
  messages delegating certain powers to a new key). This also affects
  the repairer: who gets charged when the repairer creates new shares?
* sub-accounts: allowing Alice to delegate some of her allocated storage
  space to Amy: the server doesn't care to distinguish between the two,
  but Alice and Amy do.
* Invitations, and other reciprocity/grid-building techniques
* extending storage-authority into the webapi, so multiple clients could
  share a gateway and temporarily lend their storage-authority to the
  gateway for a few operations. This will require a lot of deep
  thinking: making the WUI still feasible to use, avoiding irrevocable
  authority grants, figuring out sharing links, etc.


== What's Next ==

Once 1.9 is done, I plan to walk through the #1363 changes with somebody
(probably David-Sarah) and land them all: that's just internal cleanup,
no external behavior changes.

Then we need to make a decision about public-key signature technology,
and land the #466 work. That might mean Ed25519 (and probably adding a
dependency on python-ed25519), my current favorite. It might mean
embedding python-ecdsa, or adding ECDSA to pycryptopp. Historically,
this has been the biggest impediment to progress on #466, so we probably
need to favor whatever lets this actually-get-done.

Hopefully by that point this Accounting branch will be cleaner and ready
for landing.

Then we can work on giving server admins more control, adding some of
the Ostrom-inspired ideas, implementing payment/reciprocity systems, and
building tools for larger-scale deployments (AllMyData-style account
managers, report generators, quota enforcers, etc).


cheers,
 -Brian


[1]: http://tahoe-lafs.org/pipermail/tahoe-dev/2010-December/005748.html
[2]:
https://github.com/warner/tahoe-lafs/blob/accounting/src/allmydata/storage/accountant.py#L21
-------------- next part --------------
A non-text attachment was scrubbed...
Name: server-table.png
Type: image/png
Size: 207427 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20111011/1cb19912/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: client-table.png
Type: image/png
Size: 116737 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20111011/1cb19912/attachment-0003.png>