[tahoe-dev] Thinking about building a P2P backup system

[zooko]

On Jan 8, 2009, at 10:24 AM, Shawn Willden wrote:

> It's still 30-40 seconds per image, even if you can fill the pipe  
> completely.

Yes, there is some interesting network application engineering left  
to be done if you want smooth browsing of big files stored on tahoe.   
Your idea about storing some of the shares locally is interesting,  
plus of course we might want to do pre-fetching or thumbnails.  Hm,  
if you use Windows, you might be interested in how the tahoe-w32- 
client does these things.  Or even if you don't!

> I know that par2, which uses Reed-Solomon, is terribly slow

par2 is terribly slow.  I don't know why, but it isn't just because  
of the computational cost of Reed-Solomon.  I benchmarked par2 vs.  
zfec when I first released zfec:

http://allmydata.org/trac/zfec/browser/zfec/README.txt

"""
On my Athlon 64 2.4 GHz workstation (running Linux), the "zfec"
command-line tool encoded a 160 MB file with m=100, k=94 (about 6%
redundancy) in 3.9 seconds, where the "par2" tool encoded the file
with about 6% redundancy in 27 seconds.  zfec encoded the same file
with m=12, k=6 (100% redundancy) in 4.1 seconds, where par2 encoded it
with about 100% redundancy in 7 minutes and 56 seconds.
"""

> I just used the darcs binaries available in Ubuntu 8.10.  The  
> package version is 2.0.2-2ubuntu, so I would think I did use darcs- 
> v2, and it still took tens of minutes.

Good, you already have darcs-v2, then if you were looking at the wiki  
page today you would see that it says to use the "--lazy" option and  
to fetch from http://allmydata.org/source/tahoe/trunk-hashedformat  
instead of from http://allmydata.org/source/tahoe/trunk- 
darcs1format .  Using darcs-v2 executable, --lazy, and fetching from  
the -hashedformat repository will mean that it takes only a couple of  
minutes instead of tens of minutes.

Regards,

Zooko
---
Tahoe, the Least-Authority Filesystem -- http://allmydata.org
store your data: $10/month -- http://allmydata.com/?tracking=zsig