[volunteergrid2-l] Should we increase the node minimum capacity?

[Shawn Willden]

At present, we've specified that nodes should offer a minimum of 500 GB of
storage and a maximum of 1TB.  The reason for having an upper limit is so
we don't get into a situation where one user provides, say, 20 TB and then
expects to be able to *use* 20 TB of storage.

To see why that would be a problem, assume we had 14 nodes with 500 GB
each, all empty, and one node with 20 TB.  This means the grid has a total
of 27 TB of storage... but only 7.5 TB of that is usable. Why?  Because
that 7.5 TB would get spread evenly across all 15 nodes, at which point 14
of them would be full and one would still have 19.5 TB free -- but we don't
want to put all of our data on that one node; that's the whole point of the
grid!

Given N servers, the grid begins to "degrade" (have servers that can't
accept more data) as soon as N*s data has been stored in it, where s is the
capacity of the smallest server.  With our specified minimum size of 0.5
TB, and 14 nodes, that means our grid has a non-degraded capacity of 7 TB.
 This means that one user can "swamp" the grid by uploading 7 TB, even if
that user is being perfectly fair.

To prevent this, we put a cap of 1 TB per node (technically, nodes are
allowed to provide as much as they want, but users can't consume more than
min(1TB, node_capacity).  At that rate, 7 users maxing their usage could
swamp the grid, so it's still a potential problem, just much less likely,
which is why I originally suggested that we stick with a 2:1 max/min ratio.
 I still think that's a good idea, although our present usage is such that
we're far, far from having any problem.

However, I find that I may actually want to back up more than 1 TB (after
FEC expansion) -- but that's only possible if everyone on the grid is okay
with increasing the minimum node size.

So, what do all of you think?  Is it feasible to raise the minimum node
size to 750 GB?  Or 1 TB?

I know we actually have one node in the grid with < 500 GB, so talking
about going even higher is going to be a challenge, but I want to get a
feel for what everyone thinks about this.  This doesn't have to be an
immediate thing, either... especially given that HDD prices are spiking a
bit right now.

-- 
Shawn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20111102/45de0370/attachment.html>