turn on garbage collection by default, offer obvious deep-repair-lease, warn about unset config

[zooko]

From ​this mail to tahoe-dev from Kevin Reid:

Perhaps we should turn GC on by default; perhaps with a longer-than-30-days value and a recommendation to turn it down.

Files need regular check/repair to be reliable in the long term anyway; as part of this change there ought to be a single obvious defaults-to-deep-repair-and-add-lease operation.

Also, how about the WUI presenting warnings whenever possible (directory or file-info views) about expiring leases, with a very conservative default?

Perhaps the gateway welcome page could have "Configure this!" advice until non-defaults are set.

[warner]

I dunno, I prefer Tahoe's defaults to prefer data-retention over data-expiration (preferring durability over write-availability). Turning on GC by default is going to unpleasantly surprise some people about 30 days after they upload their data, and they're going to think that Tahoe is not a reliable storage system. I think we should write and enable tools inside the client node to automate lease-renewal (like the "repair agent" idea that we've kicked around, #483 and/or #543) and give client-side users a chance to upgrade to those versions before we change the servers to automatically delete shares.

But I certainly agree that the expiration control knobs could be more visible. Maybe we should encourage each grid to write up a "welcome page" that sets out the policies of that particular grid, and for grids which choose to use expiration, put instructions on that page (both periodic-deep-renew commands for clients, and enable-expiration settings for servers).

[zooko]

#1205 was a duplicate of part of this.

[zooko]

I, too, am fairly negative about turning on share expiry by default. Honestly, I'm a bit negative on using share expiry at all. If I upload a bunch of data to your storage servers and then I accidentally get marooned on a desert island for ten years and the lease-renewing robot that I left behind suffers fatal bitrot and stops working, then when I get back you damned well better still have the shares that I entrusted to you, because I never told you to delete them.

[gdt]

I see your point - but I don't understand how without it that a) we need leases, b) servers will do anything other than fill up and become useless, and c) how people can unlink files and cause the storage to be reclaimed.

My pubgrid server grew to capacity and became non-useful without expiration. So we need explicit refcounting on remove and accounting, at least.

[zooko]

I see your point. I think what I want is an alternative garbage collection mechanism which fails safe—if the garbage-collection mechanism keeps running, but the client who has a vested interest in preserving the shares has an extended outage, the shares are preserved.

I recently understood better why Kevin Reid and I keep having differing intuitions about what would be good defaults. We have different use cases! I tend to be thinking in terms of long-term backup. If I've uploaded some data to some servers, and then those servers get full and stop accepting new data, well then that's fine! My shares are perfectly safe (as long as the garbage collection mechanism doesn't incorrectly delete them). If the server is completely full of files, then most of them are probably, like my files, intended by their owners to just sit there from now on, so from that perspective the server is doing a great job. :-)

Kevin tends to be thinking in terms of file-sharing/hosting, where each new file is important immediately after it is uploaded and then gets less and less important over time, and is eventually deleted to make room for new files. A full server is useless for that use case.

[gdt]

Then there's the scenario in the middle: a shared grid, where I'm doing backups but the servers are full because other people are putting lots of new files on and there is no expiration.

Your alternative gc scheme sounds like the right answer. One approach would be to let node operators see how old shares are, and perhaps manually expire them (as in the lease process runs, but the server doesn't actually free unleased shares). If the node operators had a userid tie to shares (maybe not so bad), there could be a notion of "expire shares without leases if the userid that used to hold the lease has completed lease renewal runs but not renewed this". Probably we want lease group ids tied to rootcaps, not users, but perhaps this can be made to work.