[tahoe-dev] Thinking about building a P2P backup system

[Shawn Willden]

On Thursday 08 January 2009 09:17:40 am zooko wrote:
> What if the data is coming from ten different connections, each of
> which runs at about 100 kbps.  Do you think that might be
> sufficiently high bandwidth for photo sharing?

It's still 30-40 seconds per image, even if you can fill the pipe completely.  
For that matter, I did some testing last night with photos uploaded to the 
test grid, and browsing them is pretty painful even on my 6 mpbs connection -- 
and the test grid did a great job of filling it.

> By the way, there is a paper coming out in FAST '09 about the
> performance of open source software erasure codes that measures zfec
> among others.

Cool.  I'll be interested to see that.  I don't have any significant firsthand 
experience with different erasure codes.  I know that par2, which uses 
Reed-Solomon, is terribly slow, and I've read the Tornado authors' claims of 
up to four orders of magnitude speedup as compared to RS.  I've been reading 
some of the original Tornado papers and I think it would be fun to implement 
it.  I'm going to stay focused on backup, though, and not let that distract 
me.

> Have you seen this thread?  It might be a good project for you, as it
> is self-contained, requires minimal changes to the tahoe core itself,
> and is closely related to your idea about good backup:
>
> http://allmydata.org/pipermail/tahoe-dev/2008-September/000809.html

I'll take a look.

> > Okay.  I grabbed the darcs repo (dang is that sloowww! Anybody for
> > switching to git? ;-)) and I'll start from there.
>
> I updated the instructions on http://allmydata.org/trac/tahoe/wiki/
> Dev to suggest using darcs-v2 and to warn that using darcs-v1 will
> take tens of minutes for the initial get.

I just used the darcs binaries available in Ubuntu 8.10.  The package version 
is 2.0.2-2ubuntu, so I would think I did use darcs-v2, and it still took tens 
of minutes.

> I would entertain the idea of switching to git, even though I love darcs 
> and contribute to darcs and use it all the time, solely in order to be more
> friendly toward potential contributors who love git.

Well, I was just joking, but I really do like git.  It's what I use for all of 
my own work, and even when I'm forced to use SVN or CVS, I use the git 
frontends to make them much faster and friendlier.

> Start here:
>
> http://allmydata.org/trac/tahoe/wiki/Doc

Great.

> Then update the wiki and/or submit patches making it easier for the
> next person who starts there to find what they are looking for.  :-)

I'll try to do that.

        Shawn.