[volunteergrid2-l] Backing up Files

[Shawn Willden]

The best way to handle that, I think, is to do it in small bites.  The
actual data files are stored as immutable files, and immutable uploads are
idempotent... meaning that once a file is in the grid, subsequent attempts
to upload it will succeed quickly when Tahoe notices that the file is
already present.

So, pick a reasonably-sized subdir and run tahoe backup on that.  Then
another, and another, etc.  Once you've got everything uploaded, then you
can run a top-level backup.  That will be somewhat time-consuming because
your Tahoe node will still have to do all of the computations on every
file, but it'll go much faster because when it starts doing the actual
upload if it finds that there are already enough shares present it won't
bother with uploading.

After your first big top-level backup, then you can re-run the backup
whenever, and it should be pretty fast.  Tahoe backup keeps a local
database so it can recognize when files haven't change and skip them
(completely skip them, not encode-then-notice-it's-unnecessary).

On Tue, Feb 7, 2012 at 3:28 PM, Brad Rupp <bradrupp at gmail.com> wrote:

> Oh boy, I definitely have more data than can be uploaded in a few days.
>  That is part of the fun I am facing with getting this done.
>
> Brad
>
>
> On 2/7/2012 2:11 PM, Shawn Willden wrote:
>
>> For relatively small amounts of data, tahoe backup works fine.  If you
>> have more than can reasonably be uploaded in a few days, you can still
>> use tahoe backup, but it's trickier.
>>
>> On Tue, Feb 7, 2012 at 11:52 AM, Brad Rupp <bradrupp at gmail.com
>> <mailto:bradrupp at gmail.com>> wrote:
>>
>>    I am curious how everyone is backing up their data to the grid.  I
>>    assume most people are using tahoe backup, but I'm wondering if
>>    there are other solutions that have been hacked together.
>>
>>    As for me, I'm just getting started on this.  My server going up in
>>    smoke last month has caused a sense of urgency regarding backups.
>>      Of course, that is why I am still working on this a month later... :)
>>
>>    My preferred backup method would be to push a weekly snapshot to the
>>    grid.  I can live with losing a weeks worth of data and I don't need
>>    versioned backups.  Basically the last weekly snapshot is all that I
>>    need.  Now to make it happen...
>>
>>    Regards,
>>
>>    Brad
>>
>>
>>    ______________________________**___________________
>>    volunteergrid2-l mailing list
>>    volunteergrid2-l at tahoe-lafs.__**org
>>    <mailto:volunteergrid2-l@**tahoe-lafs.org<volunteergrid2-l at tahoe-lafs.org>
>> >
>>    http://tahoe-lafs.org/cgi-bin/**__mailman/listinfo/__**
>> volunteergrid2-l<http://tahoe-lafs.org/cgi-bin/__mailman/listinfo/__volunteergrid2-l>
>>    <http://tahoe-lafs.org/cgi-**bin/mailman/listinfo/**volunteergrid2-l<http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l>
>> >
>>    http://bigpig.org/twiki/bin/__**view/Main/WebHome<http://bigpig.org/twiki/bin/__view/Main/WebHome>
>>    <http://bigpig.org/twiki/bin/**view/Main/WebHome<http://bigpig.org/twiki/bin/view/Main/WebHome>
>> >
>>
>>
>>
>>
>> --
>> Shawn
>>
>>
>>
>> ______________________________**_________________
>> volunteergrid2-l mailing list
>> volunteergrid2-l at tahoe-lafs.**org <volunteergrid2-l at tahoe-lafs.org>
>> http://tahoe-lafs.org/cgi-bin/**mailman/listinfo/**volunteergrid2-l<http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l>
>> http://bigpig.org/twiki/bin/**view/Main/WebHome<http://bigpig.org/twiki/bin/view/Main/WebHome>
>>
> ______________________________**_________________
> volunteergrid2-l mailing list
> volunteergrid2-l at tahoe-lafs.**org <volunteergrid2-l at tahoe-lafs.org>
> http://tahoe-lafs.org/cgi-bin/**mailman/listinfo/**volunteergrid2-l<http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l>
> http://bigpig.org/twiki/bin/**view/Main/WebHome<http://bigpig.org/twiki/bin/view/Main/WebHome>
>



-- 
Shawn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tahoe-lafs.org/cgi-bin/mailman/private/volunteergrid2-l/attachments/20120207/d65d40fb/attachment-0001.html>