[tahoe-lafs-trac-stream] [tahoe-lafs] #1835: stop grovelling the whole storage backend looking for externally-added shares to add a lease to

[tahoe-lafs]

#1835: stop grovelling the whole storage backend looking for externally-added
shares to add a lease to
-------------------------+-------------------------------------------------
     Reporter:  zooko    |      Owner:
         Type:           |     Status:  new
  enhancement            |  Milestone:  undecided
     Priority:  normal   |    Version:  1.9.2
    Component:  code-    |   Keywords:  leases garbage-collection
  storage                |  accounting
   Resolution:           |
Launchpad Bug:           |
-------------------------+-------------------------------------------------

Comment (by davidsarah):

 Replying to [ticket:1835 zooko]:
 > Currently, storage server operators can manually add share files into
 the storage backend, such as with "mv" or "rsync" or what have you, and a
 crawler will eventually discover that share and add a lease to it.
 >
 > I propose that we stop supporting this method of installing shares. If
 we stop supporting this, that would leave three options for if you want to
 add a share to a server:
 >
 > 1. Send it through the front door — use a tool that speaks the LAFS
 protocol, connects to the storage server over a network socket, and
 delivers the share. [...]
 > 2. Copy the shares directly into place in the storage backend and then
 remove the leasedb.

 I don't like this option because it unnecessarily loses accounting
 information.

 > 3. Copy the shares into place, but don't do anything that would register
 them in the leasedb. They are now immortal, unless a client subsequently
 adds a lease to them.

 The fact that not doing anything to register the existence of the share is
 safe (doesn't lose data) is a useful property.

 Note that this would only work if a server still queries the backend for a
 share even if it does not exist in the leasedb, rather than taking the
 leasedb as authoritative.

 Perhaps if the share is ever requested, the server could then notice that
 it exists and add it to the leasedb. In that case, doing a filecheck on
 that file would be sufficient.

 > 4. Copy the shares into place and then use a newly added feature of
 storage server which tells it to notice the existence of each new share
 (by storage index). This newly added feature doesn't need to be exported
 over the network to remote foolscap clients, it could just be a "tahoe"
 command-line that connects to the storage server's local WAPI.

 Note that currently, running a WAPI is optional for storage servers.

 > 5. Copy the shares into place and then use a newly added feature of
 storage server which performs a full crawl to update the leasedb without
 first deleting it.

 Yes.

 > 4 would be a bit more efficient than 5 when used, but a lot more
 complication for the server administrator, who has to figure out how to
 call {{{tahoe add-share-to-lease-db $STORAGEINDEX}}} for each share that
 he's added, or else that share will be immortal. It is also more work for
 us to implement.

 I think the variant where requesting the share is sufficient to make the
 server notice it is simpler.

 > 5 is really simple both for us to implement and storage server operators
 to use. It is exactly like the current crawler code, except that instead
 of continuously restarting itself and going to look for new shares, it
 quiesces and doesn't restart unless the server operator invokes {{{tahoe
 resync-lease-db}}}.

 We can support that as well, subject to the caveat that the storage server
 WAPI is optional.

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1835#comment:1>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage