[tahoe-dev] Questions

[Brian Warner]

On 5/12/10 1:24 PM, Jason Wood wrote:
> 
> Ok, so it seems to do everything I need, now for a couple of questions
> about "nice to have" features... 
> 
> I believe I can set up a grid to consist of nodes on a LAN and WAN as
> part of the same storage cluster. So, if I had 3 locations each with 5
> storage nodes, could I configure the grid to ensure a file is written to
> each location so that I could handle all servers at a particular
> location going down?

Ah, no, not directly. We have a ticket about that one (#467, #302), but
it's deeper than it looks and we haven't come to a conclusion on how to
build it.

The current system will try to distribute the shares as widely as
possible, using a different pseudo-random permutation for each file, but
it is completely unaware of server properties like "location". If you
have more free servers than shares, it will only put one share on any
given server, but you might wind up with more shares in one location
than the others.

For example, if you have 15 servers in three locations A:1/2/3/4/5,
B:6/7/8/9/10, C:11/12/13/14/15, and use the default 3-of-10 encoding,
your worst case is winding up with shares on 1/2/3/4/5/6/7/8/9/10, and
not use location C at all. The most *likely* case is that you'll wind up
with 3 or 4 shares in each location, but there's nothing in the system
to enforce that: it's just shuffling all the servers into a ring,
starting at 0, and assigning shares to servers around and around the
ring until all the shares have a home.

There's some math we could do to estimate the probability of things like
this, but I'd have to dust off a stats textbook to remember what it is.
(actually, since 15-choose-10 is only 3003, I'd write a little python
program to simulate all the possibilities, and then count the results).

[brian spends 5 minutes writing the script, attached]

Ok, so the possibilities are:

 (3, 3, 4) 1500
 (2, 4, 4) 750
 (2, 3, 5) 600
 (1, 4, 5) 150
 (0, 5, 5) 3
  sum =    3003

So you've got a 50% chance of the ideal distribution, and a 1/1000
chance of the worst-case distribution.


> And finally, is it possible to modify a mutable file by "patching" it?
> So, if I have a file stored and I want to update a section of the file
> in the middle, is that possible or would be file need to be downloaded,
> patched and re-uploaded? I think I'm asking a lot here and I already
> have a plan to work around it but as the system seems to do everything
> else I need I figured it was worth asking.

Not at present. We've only implemented "Small Distributed Mutable Files"
(SDMF) so far, which have the property that the whole file must be
downloaded or uploaded at once. We have plans for "medium" MDMF files,
which will fix this. MDMF files are broken into segments (default size
is 128KiB), and you only have to replace the segments that are dirtied
by the write, so changing a single byte would only require the upload of
N/k*128KiB or about 440KiB for the default 3-of-10 encoding.

Kevan Carstensen is spending his summer implementing MDMF, thanks to the
sponsorship of Google Summer Of Code. Ticket #393 is tracking this work.

cheers,
 -Brian


#302: http://allmydata.org/trac/tahoe-lafs/ticket/302
#467: http://allmydata.org/trac/tahoe-lafs/ticket/467
#393: http://allmydata.org/trac/tahoe-lafs/ticket/393

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: sharedist.py
Url: http://allmydata.org/pipermail/tahoe-dev/attachments/20100512/76f40257/attachment.txt