[volunteergrid2-l] Backing up Files

[Brad Rupp]

Thanks Shawn.

One more question for the group.  Since tahoe backup does a versioned 
backup, is there any way to remove old files from the grid that have 
been deleted on the source?  For example, say I backup my pictures each 
night using tahoe backup.  Now, a month later, I do some house cleaning 
and remove a bunch of old pictures.  My understanding is that on the 
grid those files will not be in the current version of my backup, but 
they will be in the version from 30 days ago.  Is there a way to remove 
them so I'm not wasting space on the grid with files I no longer care about?

I know this will eventually be taken care of by the garbage collector. 
I just don't want to waste space for a year with old files.

Thanks,

Brad

On 2/7/2012 3:47 PM, Shawn Willden wrote:
> The best way to handle that, I think, is to do it in small bites.  The
> actual data files are stored as immutable files, and immutable uploads
> are idempotent... meaning that once a file is in the grid, subsequent
> attempts to upload it will succeed quickly when Tahoe notices that the
> file is already present.
>
> So, pick a reasonably-sized subdir and run tahoe backup on that.  Then
> another, and another, etc.  Once you've got everything uploaded, then
> you can run a top-level backup.  That will be somewhat time-consuming
> because your Tahoe node will still have to do all of the computations on
> every file, but it'll go much faster because when it starts doing the
> actual upload if it finds that there are already enough shares present
> it won't bother with uploading.
>
> After your first big top-level backup, then you can re-run the backup
> whenever, and it should be pretty fast.  Tahoe backup keeps a local
> database so it can recognize when files haven't change and skip them
> (completely skip them, not encode-then-notice-it's-unnecessary).
>
> On Tue, Feb 7, 2012 at 3:28 PM, Brad Rupp <bradrupp at gmail.com
> <mailto:bradrupp at gmail.com>> wrote:
>
>     Oh boy, I definitely have more data than can be uploaded in a few
>     days.  That is part of the fun I am facing with getting this done.
>
>     Brad
>
>
>     On 2/7/2012 2:11 PM, Shawn Willden wrote:
>
>         For relatively small amounts of data, tahoe backup works fine.
>           If you
>         have more than can reasonably be uploaded in a few days, you can
>         still
>         use tahoe backup, but it's trickier.
>
>         On Tue, Feb 7, 2012 at 11:52 AM, Brad Rupp <bradrupp at gmail.com
>         <mailto:bradrupp at gmail.com>
>         <mailto:bradrupp at gmail.com <mailto:bradrupp at gmail.com>>> wrote:
>
>             I am curious how everyone is backing up their data to the
>         grid.  I
>             assume most people are using tahoe backup, but I'm wondering if
>             there are other solutions that have been hacked together.
>
>             As for me, I'm just getting started on this.  My server
>         going up in
>             smoke last month has caused a sense of urgency regarding
>         backups.
>               Of course, that is why I am still working on this a month
>         later... :)
>
>             My preferred backup method would be to push a weekly
>         snapshot to the
>             grid.  I can live with losing a weeks worth of data and I
>         don't need
>             versioned backups.  Basically the last weekly snapshot is
>         all that I
>             need.  Now to make it happen...
>
>             Regards,
>
>             Brad
>
>
>             ___________________________________________________
>             volunteergrid2-l mailing list
>             volunteergrid2-l at tahoe-lafs.____org
>         <mailto:volunteergrid2-l at __tahoe-lafs.org
>         <mailto:volunteergrid2-l at tahoe-lafs.org>>
>         http://tahoe-lafs.org/cgi-bin/____mailman/listinfo/____volunteergrid2-l
>         <http://tahoe-lafs.org/cgi-bin/__mailman/listinfo/__volunteergrid2-l>
>         <http://tahoe-lafs.org/cgi-__bin/mailman/listinfo/__volunteergrid2-l
>         <http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l>>
>         http://bigpig.org/twiki/bin/____view/Main/WebHome
>         <http://bigpig.org/twiki/bin/__view/Main/WebHome>
>         <http://bigpig.org/twiki/bin/__view/Main/WebHome
>         <http://bigpig.org/twiki/bin/view/Main/WebHome>>
>
>
>
>
>         --
>         Shawn
>
>
>
>         _________________________________________________
>         volunteergrid2-l mailing list
>         volunteergrid2-l at tahoe-lafs.__org
>         <mailto:volunteergrid2-l at tahoe-lafs.org>
>         http://tahoe-lafs.org/cgi-bin/__mailman/listinfo/__volunteergrid2-l
>         <http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l>
>         http://bigpig.org/twiki/bin/__view/Main/WebHome
>         <http://bigpig.org/twiki/bin/view/Main/WebHome>
>
>     _________________________________________________
>     volunteergrid2-l mailing list
>     volunteergrid2-l at tahoe-lafs.__org
>     <mailto:volunteergrid2-l at tahoe-lafs.org>
>     http://tahoe-lafs.org/cgi-bin/__mailman/listinfo/__volunteergrid2-l
>     <http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l>
>     http://bigpig.org/twiki/bin/__view/Main/WebHome
>     <http://bigpig.org/twiki/bin/view/Main/WebHome>
>
>
>
>
> --
> Shawn
>
>
> _______________________________________________
> volunteergrid2-l mailing list
> volunteergrid2-l at tahoe-lafs.org
> http://tahoe-lafs.org/cgi-bin/mailman/listinfo/volunteergrid2-l
> http://bigpig.org/twiki/bin/view/Main/WebHome