[volunteergrid2-l] Thoughts on catastrophic events and garbage collection/expiration

[Shawn Willden]

On Wed, Feb 2, 2011 at 2:55 PM, Jody Harris <jharris at harrisdev.com> wrote:

> If this happens with 3 members/year, that's a 2TB drive a year buried under
> dead weight.... and it's going to be cumulative. [...] 140GB/yr, expanded
> into ~300GB/year of accumulated and dead shares.
>
> ..... which in the end may not be that big of a deal.
>

If storage capacities continue to follow the curve that they have -- and we
know that data volumes tend to track pretty closely to available storage --
those sources of dead shares may not matter at all.  One way to think of it
is that storage capacity and demand remains constant, but the space consumed
by old dead stuff decreases by 50% every 1.5 years.  This exponential
falloff of wasted space may mean that the waste never becomes relevant.

Or maybe usage will climb faster than storage, or storage capacity growth
will tail off or other users will have much more dynamic data sets (and the
bandwidth to push them into the grid)... who knows?

But my thought is that until we actually see that storage is getting low,
why not just let the cruft accumulate?  If we get to that point, we can just
get confirmation from everyone on the list that they've recently renewed the
leases all the shares they care about, and then turn on expiration.


> I guess one think that would help would be a peer-reviewed drive upgrade
> strategy. That way, when I replace my 500 LVM in (say) 6 months with a 2 TB
> drive, if we have <20 nodes, I might move all of the shares to the new
> drive, but if we had >20 nodes, we might decide to just let the shares die
> with the old drive, which would give us a somewhat graceful way to retire
> old, de-linked shares, and active capabilities would be repaired within a
> month.
>

My immediate reaction to this idea is that I don't like it.  Deliberately
allowing shares to be lost when they could be retained rubs me the wrong
way... but it really shouldn't be a big deal.  I have to think about that.

-- 
Shawn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20110202/df86fb9b/attachment-0001.html>