[volunteergrid2-l] Should we increase the node minimum capacity?

[Brad Rupp]

I'm good with 1TB per node, but that is easy for me to say as both of my 
nodes already have 1TB each.  Before we do this, we definitely need to 
address any nodes that are < 500GB.

Brad

On 11/2/2011 2:08 PM, Shawn Willden wrote:
> At present, we've specified that nodes should offer a minimum of 500 GB
> of storage and a maximum of 1TB.  The reason for having an upper limit
> is so we don't get into a situation where one user provides, say, 20 TB
> and then expects to be able to *use* 20 TB of storage.
>
> To see why that would be a problem, assume we had 14 nodes with 500 GB
> each, all empty, and one node with 20 TB.  This means the grid has a
> total of 27 TB of storage... but only 7.5 TB of that is usable. Why?
>   Because that 7.5 TB would get spread evenly across all 15 nodes, at
> which point 14 of them would be full and one would still have 19.5 TB
> free -- but we don't want to put all of our data on that one node;
> that's the whole point of the grid!
>
> Given N servers, the grid begins to "degrade" (have servers that can't
> accept more data) as soon as N*s data has been stored in it, where s is
> the capacity of the smallest server.  With our specified minimum size of
> 0.5 TB, and 14 nodes, that means our grid has a non-degraded capacity of
> 7 TB.  This means that one user can "swamp" the grid by uploading 7 TB,
> even if that user is being perfectly fair.
>
> To prevent this, we put a cap of 1 TB per node (technically, nodes are
> allowed to provide as much as they want, but users can't consume more
> than min(1TB, node_capacity).  At that rate, 7 users maxing their usage
> could swamp the grid, so it's still a potential problem, just much less
> likely, which is why I originally suggested that we stick with a 2:1
> max/min ratio.  I still think that's a good idea, although our present
> usage is such that we're far, far from having any problem.
>
> However, I find that I may actually want to back up more than 1 TB
> (after FEC expansion) -- but that's only possible if everyone on the
> grid is okay with increasing the minimum node size.
>
> So, what do all of you think?  Is it feasible to raise the minimum node
> size to 750 GB?  Or 1 TB?
>
> I know we actually have one node in the grid with < 500 GB, so talking
> about going even higher is going to be a challenge, but I want to get a
> feel for what everyone thinks about this.  This doesn't have to be an
> immediate thing, either... especially given that HDD prices are spiking
> a bit right now.
>
> --
> Shawn
>
>
> _______________________________________________
> volunteergrid2-l mailing list
> volunteergrid2-l at tahoe-lafs.org
> http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
> http://bigpig.org/twiki/bin/view/Main/WebHome