[volunteergrid2-l] Thoughts on catastrophic events and garbage collection/expiration

[Jody Harris]

On Wed, Feb 2, 2011 at 2:33 PM, Shawn Willden <shawn at willden.org> wrote:

> On Wed, Feb 2, 2011 at 2:28 PM, Jody Harris <jharris at harrisdev.com> wrote:
>
>> BESIDES, which, you're supposed to be arguing with me over this!
>>
>
> Okay, I'll argue.
>
> I think there may never be any reason to turn it on :-)
>
> At least for me, very little of what I back up is dynamic or transient.  I
> have a slowly-growing pile of data that I want to keep forever.  Dirnodes
> will become obsoleted over time, but they comprise such a miniscule portion
> of the total data that they're basically lost in the noise.
>
> Nice.

However, the nature of this grid will mean that there will be people coming
in and leaving over time. If I were to backup 300 GB of photos and other
junk to the grid, then decide I don't want to play with you any more, there
is going to be 500 GB of data setting on other people's drives until those
units fail in such a way that the data doesn't get moved to the replacement
drive first. If this happens with 3 members/year, that's a 2TB drive a year
buried under dead weight.... and it's going to be cumulative.

In the case of my server backups (mostly databases that change daily), we're
looking at ~ 40MB/day, ~14GB/year of accumulated immutable files stacking
up. If 10 people use it for similar purposes, that's 140GB/yr, expanded into
~300GB/year of accumulated and dead shares.

..... which in the end may not be that big of a deal.

I guess one think that would help would be a peer-reviewed drive upgrade
strategy. That way, when I replace my 500 LVM in (say) 6 months with a 2 TB
drive, if we have <20 nodes, I might move all of the shares to the new
drive, but if we had >20 nodes, we might decide to just let the shares die
with the old drive, which would give us a somewhat graceful way to retire
old, de-linked shares, and active capabilities would be repaired within a
month.

hrm....

j
----
- Think carefully.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20110202/ef025eac/attachment.html>