[volunteergrid2-l] Backing up Files

[Steve Dodson]

Thank you Shawn - this is exactly the sort of information I was looking 
for (I was in mid-reply to Brad whenever this came through).

On 02/07/2012 03:47 PM, Shawn Willden wrote:
> The best way to handle that, I think, is to do it in small bites.  The
> actual data files are stored as immutable files, and immutable uploads
> are idempotent... meaning that once a file is in the grid, subsequent
> attempts to upload it will succeed quickly when Tahoe notices that the
> file is already present.
>
> So, pick a reasonably-sized subdir and run tahoe backup on that.  Then
> another, and another, etc.  Once you've got everything uploaded, then
> you can run a top-level backup.  That will be somewhat time-consuming
> because your Tahoe node will still have to do all of the computations on
> every file, but it'll go much faster because when it starts doing the
> actual upload if it finds that there are already enough shares present
> it won't bother with uploading.
>
> After your first big top-level backup, then you can re-run the backup
> whenever, and it should be pretty fast.  Tahoe backup keeps a local
> database so it can recognize when files haven't change and skip them
> (completely skip them, not encode-then-notice-it's-unnecessary).
>
> On Tue, Feb 7, 2012 at 3:28 PM, Brad Rupp <bradrupp at gmail.com
> <mailto:bradrupp at gmail.com>> wrote:
>
>     Oh boy, I definitely have more data than can be uploaded in a few
>     days.  That is part of the fun I am facing with getting this done.
>
>     Brad
>
>
>     On 2/7/2012 2:11 PM, Shawn Willden wrote:
>
>         For relatively small amounts of data, tahoe backup works fine.
>           If you
>         have more than can reasonably be uploaded in a few days, you can
>         still
>         use tahoe backup, but it's trickier.
>
>         On Tue, Feb 7, 2012 at 11:52 AM, Brad Rupp <bradrupp at gmail.com
>         <mailto:bradrupp at gmail.com>
>         <mailto:bradrupp at gmail.com <mailto:bradrupp at gmail.com>>> wrote:
>
>             I am curious how everyone is backing up their data to the
>         grid.  I
>             assume most people are using tahoe backup, but I'm wondering if
>             there are other solutions that have been hacked together.
>
>             As for me, I'm just getting started on this.  My server
>         going up in
>             smoke last month has caused a sense of urgency regarding
>         backups.
>               Of course, that is why I am still working on this a month
>         later... :)
>
>             My preferred backup method would be to push a weekly
>         snapshot to the
>             grid.  I can live with losing a weeks worth of data and I
>         don't need
>             versioned backups.  Basically the last weekly snapshot is
>         all that I
>             need.  Now to make it happen...
>
>             Regards,
>
>             Brad
>
>
>             ___________________________________________________
>             volunteergrid2-l mailing list
>             volunteergrid2-l at tahoe-lafs.____org
>         <mailto:volunteergrid2-l at __tahoe-lafs.org
>         <mailto:volunteergrid2-l at tahoe-lafs.org>>
>         http://tahoe-lafs.org/cgi-bin/____mailman/listinfo/____volunteergrid2-l
>         <http://tahoe-lafs.org/cgi-bin/__mailman/listinfo/__volunteergrid2-l>
>         <http://tahoe-lafs.org/cgi-__bin/mailman/listinfo/__volunteergrid2-l
>         <http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l>>
>         http://bigpig.org/twiki/bin/____view/Main/WebHome
>         <http://bigpig.org/twiki/bin/__view/Main/WebHome>
>         <http://bigpig.org/twiki/bin/__view/Main/WebHome
>         <http://bigpig.org/twiki/bin/view/Main/WebHome>>
>
>
>
>
>         --
>         Shawn
>
>
>
>         _________________________________________________
>         volunteergrid2-l mailing list
>         volunteergrid2-l at tahoe-lafs.__org
>         <mailto:volunteergrid2-l at tahoe-lafs.org>
>         http://tahoe-lafs.org/cgi-bin/__mailman/listinfo/__volunteergrid2-l
>         <http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l>
>         http://bigpig.org/twiki/bin/__view/Main/WebHome
>         <http://bigpig.org/twiki/bin/view/Main/WebHome>
>
>     _________________________________________________
>     volunteergrid2-l mailing list
>     volunteergrid2-l at tahoe-lafs.__org
>     <mailto:volunteergrid2-l at tahoe-lafs.org>
>     http://tahoe-lafs.org/cgi-bin/__mailman/listinfo/__volunteergrid2-l
>     <http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l>
>     http://bigpig.org/twiki/bin/__view/Main/WebHome
>     <http://bigpig.org/twiki/bin/view/Main/WebHome>
>
>
>
>
> --
> Shawn
>
>
>
> _______________________________________________
> volunteergrid2-l mailing list
> volunteergrid2-l at tahoe-lafs.org
> http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
> http://bigpig.org/twiki/bin/view/Main/WebHome

-- 

soli Deo gloria