[tahoe-lafs-trac-stream] [tahoe-lafs] #1834: stop using share crawler for anything except constructing a leasedb

[tahoe-lafs]

#1834: stop using share crawler for anything except constructing a leasedb
-------------------------------------------------+-------------------------
 Reporter:  zooko                                |          Owner:
     Type:  defect                               |         Status:  new
 Priority:  normal                               |      Milestone:
Component:  code-storage                         |  undecided
 Keywords:  leases garbage-collection            |        Version:  1.9.2
  accounting                                     |  Launchpad Bug:
-------------------------------------------------+-------------------------
 I think we should stop using a "share crawler" — a long-running,
 persistent, duty-cycle-limited process that visits every share held by a
 storage server — for everything that we can.

 And, I think that the only thing that we can't do in a different way is:
 construct a leasedb when we are first upgrading the server to a leasedb-
 capable version, or the leasedb has been lost or corrupted.

 Here are the other things that are currently done by crawlers and how I
 think they should be done differently:

 * Updating and/or checking the leases on shares to see if they have
 expired;

 On David-Sarah's 666-accounting branch, this is now done for all shares by
 a single, synchronous command/query to leasedb. (#666)

 * Delete shares that have lost all their leases (by cancellation or
 expiry);

 I propose that this be done instead by the storage server maintaining a
 persistent set of shares to be deleted. When lease-updating step (which,
 in #666, is synchronous and fast) has identified a share that has no more
 leases, the share's id gets added to the persistent set of shares to
 delete. A long-running, persistent, duty-cycle-limited processes deletes
 those shares from the backend and removes their ids from the set of
 shares-to-delete. This is cleaner and more efficient than using a crawler,
 which has to visit ''all'' shares and which never stops twitching, since
 this has to visit only shares that have been marked as to-delete, and it
 quiesces when there is nothing to delete. (#1833)

 * Discover newly added shares that the operator copied into the backend
 without notifying the storage server;

 I propose that we stop supporting this method of moving shares around. If
 we stop supporting this, that would leave two options for if you want to
 add a share to a server:

 1. Send it through the front door —

 I'm going to create this ticket in order to get a ticket number (probably
 #1834) that other tickets can reference, then come back and write more of
 this Description...

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1834>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage