source: trunk/docs/garbage-collection.rst

Last change on this file was 4723af6, checked in by Brian Warner <warner@…>, at 2017-06-06T10:20:49Z

LAFS now stands for "Least-Authority File Store" rather than "Least-Authority File System". refs #2345

Signed-off-by: Daira Hopwood <daira@…>

  • Property mode set to 100644
File size: 14.0 KB
1.. -*- coding: utf-8-with-signature -*-
4Garbage Collection in Tahoe
71. `Overview`_
82. `Client-side Renewal`_
93. `Server Side Expiration`_
104. `Expiration Progress`_
115. `Future Directions`_
16When a file or directory in a Tahoe-LAFS file store is no longer referenced,
17the space that its shares occupied on each storage server can be freed,
18making room for other shares. Tahoe currently uses a garbage collection
19("GC") mechanism to implement this space-reclamation process. Each share has
20one or more "leases", which are managed by clients who want the
21file/directory to be retained. The storage server accepts each share for a
22pre-defined period of time, and is allowed to delete the share if all of the
23leases expire.
25Garbage collection is not enabled by default: storage servers will not delete
26shares without being explicitly configured to do so. When GC is enabled,
27clients are responsible for renewing their leases on a periodic basis at
28least frequently enough to prevent any of the leases from expiring before the
29next renewal pass.
31There are several tradeoffs to be considered when choosing the renewal timer
32and the lease duration, and there is no single optimal pair of values. See
33the following diagram to get an idea of the tradeoffs involved:
35.. image:: lease-tradeoffs.svg
38If lease renewal occurs quickly and with 100% reliability, than any renewal
39time that is shorter than the lease duration will suffice, but a larger ratio
40of duration-over-renewal-time will be more robust in the face of occasional
41delays or failures.
43The current recommended values for a small Tahoe grid are to renew the leases
44once a week, and give each lease a duration of 31 days. In the current
45release, there is not yet a way to create a lease with a different duration,
46but the server can use the ``expire.override_lease_duration`` configuration
47setting to increase or decrease the effective duration (when the lease is
48processed) to something other than 31 days.
50Renewing leases can be expected to take about one second per file/directory,
51depending upon the number of servers and the network speeds involved.
55Client-side Renewal
58If all of the files and directories which you care about are reachable from a
59single starting point (usually referred to as a "rootcap"), and you store
60that rootcap as an alias (via "``tahoe create-alias``" for example), then the
61simplest way to renew these leases is with the following CLI command::
63  tahoe deep-check --add-lease ALIAS:
65This will recursively walk every directory under the given alias and renew
66the leases on all files and directories. (You may want to add a ``--repair``
67flag to perform repair at the same time.) Simply run this command once a week
68(or whatever other renewal period your grid recommends) and make sure it
69completes successfully. As a side effect, a manifest of all unique files and
70directories will be emitted to stdout, as well as a summary of file sizes and
71counts. It may be useful to track these statistics over time.
73Note that newly uploaded files (and newly created directories) get an initial
74lease too: the ``--add-lease`` process is only needed to ensure that all
75older objects have up-to-date leases on them.
77A separate "rebalancing manager/service" is also planned -- see ticket
78`#543`_. The exact details of what this service will do are not settled, but
79it is likely to work by acquiring manifests from rootcaps on a periodic
80basis, keeping track of checker results, managing lease-addition, and
81prioritizing repair and rebalancing of shares. Eventually it may use multiple
82worker nodes to perform these jobs in parallel.
84.. _#543:
87Server Side Expiration
90Expiration must be explicitly enabled on each storage server, since the
91default behavior is to never expire shares. Expiration is enabled by adding
92config keys to the ``[storage]`` section of the ``tahoe.cfg`` file (as described
93below) and restarting the server node.
95Each lease has two parameters: a create/renew timestamp and a duration. The
96timestamp is updated when the share is first uploaded (i.e. the file or
97directory is created), and updated again each time the lease is renewed (i.e.
98"``tahoe check --add-lease``" is performed). The duration is currently fixed
99at 31 days, and the "nominal lease expiration time" is simply $duration
100seconds after the $create_renew timestamp. (In a future release of Tahoe, the
101client will get to request a specific duration, and the server will accept or
102reject the request depending upon its local configuration, so that servers
103can achieve better control over their storage obligations.)
105The lease-expiration code has two modes of operation. The first is age-based:
106leases are expired when their age is greater than their duration. This is the
107preferred mode: as long as clients consistently update their leases on a
108periodic basis, and that period is shorter than the lease duration, then all
109active files and directories will be preserved, and the garbage will
110collected in a timely fashion.
112Since there is not yet a way for clients to request a lease duration of other
113than 31 days, there is a ``tahoe.cfg`` setting to override the duration of all
114leases. If, for example, this alternative duration is set to 60 days, then
115clients could safely renew their leases with an add-lease operation perhaps
116once every 50 days: even though nominally their leases would expire 31 days
117after the renewal, the server would not actually expire the leases until 60
118days after renewal.
120The other mode is an absolute-date-cutoff: it compares the create/renew
121timestamp against some absolute date, and expires any lease which was not
122created or renewed since the cutoff date. If all clients have performed an
123add-lease some time after March 20th, you could tell the storage server to
124expire all leases that were created or last renewed on March 19th or earlier.
125This is most useful if you have a manual (non-periodic) add-lease process.
126Note that there is not much point to running a storage server in this mode
127for a long period of time: once the lease-checker has examined all shares and
128expired whatever it is going to expire, the second and subsequent passes are
129not going to find any new leases to remove.
131The ``tahoe.cfg`` file uses the following keys to control lease expiration:
135``expire.enabled = (boolean, optional)``
137    If this is ``True``, the storage server will delete shares on which all
138    leases have expired. Other controls dictate when leases are considered to
139    have expired. The default is ``False``.
141``expire.mode = (string, "age" or "cutoff-date", required if expiration enabled)``
143    If this string is "age", the age-based expiration scheme is used, and the
144    ``expire.override_lease_duration`` setting can be provided to influence the
145    lease ages. If it is "cutoff-date", the absolute-date-cutoff mode is
146    used, and the ``expire.cutoff_date`` setting must be provided to specify
147    the cutoff date. The mode setting currently has no default: you must
148    provide a value.
150    In a future release, this setting is likely to default to "age", but in
151    this release it was deemed safer to require an explicit mode
152    specification.
154``expire.override_lease_duration = (duration string, optional)``
156    When age-based expiration is in use, a lease will be expired if its
157    ``lease.create_renew`` timestamp plus its ``lease.duration`` time is
158    earlier/older than the current time. This key, if present, overrides the
159    duration value for all leases, changing the algorithm from::
161      if (lease.create_renew_timestamp + lease.duration) < now:
162          expire_lease()
164    to::
166      if (lease.create_renew_timestamp + override_lease_duration) < now:
167          expire_lease()
169    The value of this setting is a "duration string", which is a number of
170    days, months, or years, followed by a units suffix, and optionally
171    separated by a space, such as one of the following::
173      7days
174      31day
175      60 days
176      2mo
177      3 month
178      12 months
179      2years
181    This key is meant to compensate for the fact that clients do not yet have
182    the ability to ask for leases that last longer than 31 days. A grid which
183    wants to use faster or slower GC than a 31-day lease timer permits can
184    use this parameter to implement it.
186    This key is only valid when age-based expiration is in use (i.e. when
187    ``expire.mode = age`` is used). It will be rejected if cutoff-date
188    expiration is in use.
190``expire.cutoff_date = (date string, required if mode=cutoff-date)``
192    When cutoff-date expiration is in use, a lease will be expired if its
193    create/renew timestamp is older than the cutoff date. This string will be
194    a date in the following format::
196      2009-01-16   (January 16th, 2009)
197      2008-02-02
198      2007-12-25
200    The actual cutoff time shall be midnight UTC at the beginning of the
201    given day. Lease timers should naturally be generous enough to not depend
202    upon differences in timezone: there should be at least a few days between
203    the last renewal time and the cutoff date.
205    This key is only valid when cutoff-based expiration is in use (i.e. when
206    "expire.mode = cutoff-date"). It will be rejected if age-based expiration
207    is in use.
209  expire.immutable = (boolean, optional)
211    If this is False, then immutable shares will never be deleted, even if
212    their leases have expired. This can be used in special situations to
213    perform GC on mutable files but not immutable ones. The default is True.
215  expire.mutable = (boolean, optional)
217    If this is False, then mutable shares will never be deleted, even if
218    their leases have expired. This can be used in special situations to
219    perform GC on immutable files but not mutable ones. The default is True.
221Expiration Progress
224In the current release, leases are stored as metadata in each share file, and
225no separate database is maintained. As a result, checking and expiring leases
226on a large server may require multiple reads from each of several million
227share files. This process can take a long time and be very disk-intensive, so
228a "share crawler" is used. The crawler limits the amount of time looking at
229shares to a reasonable percentage of the storage server's overall usage: by
230default it uses no more than 10% CPU, and yields to other code after 100ms. A
231typical server with 1.1M shares was observed to take 3.5 days to perform this
232rate-limited crawl through the whole set of shares, with expiration disabled.
233It is expected to take perhaps 4 or 5 days to do the crawl with expiration
234turned on.
236The crawler's status is displayed on the "Storage Server Status Page", a web
237page dedicated to the storage server. This page resides at $NODEURL/storage,
238and there is a link to it from the front "welcome" page. The "Lease
239Expiration crawler" section of the status page shows the progress of the
240current crawler cycle, expected completion time, amount of space recovered,
241and details of how many shares have been examined.
243The crawler's state is persistent: restarting the node will not cause it to
244lose significant progress. The state file is located in two files
245($BASEDIR/storage/lease_checker.state and lease_checker.history), and the
246crawler can be forcibly reset by stopping the node, deleting these two files,
247then restarting the node.
249Future Directions
252Tahoe's GC mechanism is undergoing significant changes. The global
253mark-and-sweep garbage-collection scheme can require considerable network
254traffic for large grids, interfering with the bandwidth available for regular
255uploads and downloads (and for non-Tahoe users of the network).
257A preferable method might be to have a timer-per-client instead of a
258timer-per-lease: the leases would not be expired until/unless the client had
259not checked in with the server for a pre-determined duration. This would
260reduce the network traffic considerably (one message per week instead of
261thousands), but retain the same general failure characteristics.
263In addition, using timers is not fail-safe (from the client's point of view),
264in that a client which leaves the network for an extended period of time may
265return to discover that all of their files have been garbage-collected. (It
266*is* fail-safe from the server's point of view, in that a server is not
267obligated to provide disk space in perpetuity to an unresponsive client). It
268may be useful to create a "renewal agent" to which a client can pass a list
269of renewal-caps: the agent then takes the responsibility for keeping these
270leases renewed, so the client can go offline safely. Of course, this requires
271a certain amount of coordination: the renewal agent should not be keeping
272files alive that the client has actually deleted. The client can send the
273renewal-agent a manifest of renewal caps, and each new manifest should
274replace the previous set.
276The GC mechanism is also not immediate: a client which deletes a file will
277nevertheless be consuming extra disk space (and might be charged or otherwise
278held accountable for it) until the ex-file's leases finally expire on their
281In the current release, these leases are each associated with a single "node
282secret" (stored in $BASEDIR/private/secret), which is used to generate
283renewal-secrets for each lease. Two nodes with different secrets
284will produce separate leases, and will not be able to renew each
285others' leases.
287Once the Accounting project is in place, leases will be scoped by a
288sub-delegatable "account id" instead of a node secret, so clients will be able
289to manage multiple leases per file. In addition, servers will be able to
290identify which shares are leased by which clients, so that clients can safely
291reconcile their idea of which files/directories are active against the
292server's list, and explicitly cancel leases on objects that aren't on the
293active list.
295By reducing the size of the "lease scope", the coordination problem is made
296easier. In general, mark-and-sweep is easier to implement (it requires mere
297vigilance, rather than coordination), so unless the space used by deleted
298files is not expiring fast enough, the renew/expire timed lease approach is
Note: See TracBrowser for help on using the repository browser.