Automated migration of shares between storage servers

[kpreid]

There should be a way to have a storage server automatedly move some or all of its shares to other storage servers. Right now this can be done only as a tedious manual process.

Use cases:

1. A storage server is being decommissioned; the shares would otherwise be entirely lost.
2. A storage server needs some disk space freed up for other purposes.

If it finds that another storage server has the same share (can be caused by repair), it could just be deleted locally.

Refinement:

Furthermore, since the storage server knows the server selection algorithm, it can choose to distribute particular shares to storage servers which occur early in the permuted peer order, so that the time to retrieve the shares will be better on average than it would have been if all this server's shares were moved uniformly to an arbitrarily-selected server.

In fact, if this cleverness is implemented it might be worth doing under normal conditions, to redistribute shares uploaded to the "wrong" peers because the right (early-searched) ones were unreachable at the time. (The interaction of this with #573 would need consideration.)

[warner]

This is closely related to #699 (and the other tickets that one references). Each storage server should know about the full server list, and peer-selection is a function of the server list and the storage-index, so all servers should be able to independently come up with the "right places" for a given share, and could act on their own volition to move the share into the right place (making life easier for the eventual downloader).

Ideally, a really lazy uploader who dumped all N shares on a single server should find that, eventually, their shares were redistributed to the "right" places.

There are a few wrinkles:

• Accounting: once implemented, the client will use some authority to get server A to accept their share. They may also have authority for server B, however they are unlikely to think to grant this authority to A ahead of time (since they won't know where the share might get moved to in the future, as new servers appear and old ones disappear). So A may have no way to convince B to hold this share. We could imagine some sort of swap, whereby A takes on shares from B of equal size, but this gets complicated and fails to handle some of the use cases (i.e. where A is being decomissioned or downsized).
• mutable write-enablers: each mutable share includes a per-slot-per-server secret, shared with holders of the writecap, which enables modifications to the contents. Moving such a share from one server to another will invalidate the write-enabler secret. I've written some notes on how we might update the write-enabler safely, but it's non-trivial (#489 has some details).
• leases: the new share needs to be retained for at least the same period of time as the old share. The most robust approach would be to migrate leases from the old share to the new one. But this involves shared-secrets too. Another approach would be to have the old server take out its own lease on the new share, and rely upon the expected usual periodic add-lease process to eventually establish proper leases on the new share.

But, in general, I like the idea, and it'd be nice if servers were involved in an automatic maintenance process.

[davidsarah]

Closing #481 (building some share-migration tools) as a duplicate of this. Its description was:

Zandr and I were talking about what sorts of tools we'd like to have available when it comes time to move shares from one disk to another.

The Repairer is of course the first priority, and should be able to handle share loss, but there are some techniques we might use to make things more efficient: using shares that already exist instead of generating new ones.

If we have a large disk full of shares that has some problems (bad blocks, etc), we should be able to dd or scp off the shares to another system. This wants a tool that will try to read a file (skipping it if we get io errors), verify as much of it as we can (seeing if the UEB hash matches), then sending it over the network to somewhere else.

If a disk is starting to fail (we've seen SMART statistics, or we're starting to see hash failures in the shares we return, etc), then we might want to kick the disk into "abandon ship" mode: get all shares off the disk (and onto better ones) as quickly as possible. The server could do the peer-selection work and ask around and find the "right" server for each share (i.e. the first one in the permuted order that doesn't already have a share), or it could just fling them to a "lifeboat" node and leave the peer-selection work until later.

Repair nodes should have a directory where we can dump shares that came from other servers: the repair node should treat that directory as a work queue, and it should find a home for each one (or discard it as a duplicate). The repair node needs to be careful to not treat abandon-ship nodes as suitable targets, so we can avoid putting shares back on the server that was trying to get rid of them.

It might also be useful to split up a storage server, or to take a functional server and export half of its shares in a kind of rebalancing step.

[Lcstyle]

Looking at #671 and I've suggested that this functionality would also address the issue of storage nodes that were looking to either shrink their offered storage capacity or shutdown altogether.