[volunteergrid2-l] speeds again, and missing files

[Shawn Willden]

On Tue, Apr 17, 2012 at 6:58 PM, erpo41 at gmail.com <erpo41 at gmail.com> wrote:

> Does 1.6kBps sound about right?
>

No, that's horrible.  I've been getting around 250 KBps.


> Also, I've been using the command "tahoe backup backup-T61p tahoe:". When
> I use the WUI to navigate to the writecap associated with tahoe:, it shows
> the only entry in the directory is another directory called Archive. The
> Archive directory is empty, even though I've been uploading files
> (according to upload activity log). What gives?
>

Because tahoe backup creates the directory entries as immutable files, it
can't create them until after it knows what needs to be in them, which
means not until it's uploaded the data.  This means that if you have a lot
to upload and a short GC timeout, you can find yourself unable to ever
complete your backup because the shares get GC'd before they get added to a
directory tree so their leases can be renewed!

The solution, if you have a lot to back up, is to back up smaller
subdirectories.  Because immutable uploads are idempotent, once you've
backed up all your subdirectories you can back up the top level directory
and it will be fast.

This, by the way, is one of the problems that my GridBackup tool was
intended to solve.  It is designed to separate "backup" from "upload", so
you can continue running backups even while your data is trickling out to
the grid (and there's a local backup for data that hasn't yet made it to
the grid).  Unfortunately, I didn't finish it and now no longer have time
to work on it :-(

-- 
Shawn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20120417/424e0bc5/attachment.html>