[tahoe-dev] "tahoe stats" command

[zooko]

Dear Andrej Falout:

On Feb 10, 2009, at 2:23 AM, Andrej Falout wrote:

> Then the download appears to start, there is a lot of network  
> traffic, but in 20 minutes not a single file appears in '/tmp/ 
> Pictures-RESTORE'.
>
> At which point I abort it. And read the 1.3 release announcement.  
> And wonder.

Thank you for the detailed reports and for the criticism.  If  
everyone took their mother's advice and said only the nice things  
while politely ignoring the problems, it would be harder to  
improve.  ;-)

By pressing ahead with the 1.3.0 release I do not mean to imply that  
your performance problems are unimportant.  Performance is, of  
course, of the essence.

But my current plan with tahoe-1.3.0 is just to make sure that it  
contains as few unknown bugs as possible, and is released as soon as  
possible.  It has been 7 months since the last stable release of  
tahoe (v1.2.0).  That's far too long.

Hopefully tahoe-1.3.0 will provide a "known quantity" -- a stable  
release which everyone can rely on and which we can use as a starting  
point for more improvement.

Now, about your actual performance isses, it is hard for me to  
understand the big picture.  I think that some of your earlier  
measurements turned out to just be that your upstream bandwidth isn't  
big enough to upload your whole collection quickly (no-one's is, it  
seems).  However, other of your measurements seem to indicate  
interesting underlying issues.  20 minutes without visible progress  
on "cp -r" is obviously (?) a flaw in the design or the UI of "tahoe  
cp".  Likewise your observation that many small files is much slower  
(by a factor of 8) than using duplicity clearly points to an issue in  
the way "tahoe cp" handles multiple files.  Probably it is simply  
that it doesn't do pipelining of uploads -- it uploads one file,  
waits until that one has finished uploading, and then uploads the  
next.  Let's see...  Yep:

See copy_files_to_target() --

http://allmydata.org/trac/tahoe/browser/src/allmydata/scripts/ 
tahoe_cp.py?rev=3380#L660

Hopefully someone else with more knowledge and experience than me can  
figure out why your "tahoe cp" to *download* didn't show any  
progress.  That is obviously bad behavior, but I don't know whether  
it is known bad behavior or previously-unknown bad behavior.

Thanks!

Regards,

Zooko