[tahoe-dev] Tahoe as glue filesystem

[Duncan McGreggor]

On Mon, Jul 7, 2008 at 1:24 PM, Brian Warner <warner-tahoe at allmydata.com> wrote:
>> Man, it was great to hear you ask this question. The possibility of
>> this for use in small, distributed, diskless devices is what most
>> recently prompted me to take another look at tahoe. Of course I'm
>> staying for the party anyway, but I am *deeply* interested in a
>> memory- only option for tahoe.
>
> Hm, I suppose it depends upon how much storage you have to work with. RAM or
> flash? Swapping out the allmydata.storage module for something that keeps all
> shares in a dictionary instead of in disk would be pretty easy.. we actually
> have some test cases that do just that (to test upload/download code in
> isolation from the storage code).

Sweet. That's really good to know.

> But I think it would be hard to achieve
> good reliability if your share integrity depends upon continuous power,
> especially if the nodes are small (and likely to be portable or
> battery-powered).

Well, I'm really thinking about the cases where the node counts start
getting higher (100s). For this to be practical, the repairer that you
mention below would have to be running. At that point, I think you
could define power as continuous given that the minimal set of hosts
was not without power. I would imaging that the minimal set would need
to satisfy the following constraints: so as long as the number of
hosts dropped didn't exceed the number of hosts added, the hosts
didn't drop at a rate higher than the time needed replicate the data,
and that the total number of active hosts were capable of reproducing
all data.

>> As a follow-up to this question, Zooko (and I've done no background reading
>> on this), is there any data replication in tahoe? If a host goes down and
>> doesn't come back up, are file parts redistributed to the surviving nodes?
>
> Not yet, but I'm starting work today on the "Repairer", which will take the
> remaining shares of a file and produce enough new ones to bring it back up to
> full-strength. There remains a question of when exactly to trigger this, but
> the end result would be a lot like what you're talking about: surviving (or
> brand-new) nodes wind up with parts of files that they didn't have before.

Brain,

That's exciting! I'm looking forward to hearing more about this as you
develop it. Please feel free to share progress, thoughts, ideas, etc.
on an on-going basis with the list, if you are so inclined :-)

d