[tahoe-dev] lafs-backup-tool, an alternative to "tahoe backup"

[Mike Kazantsev]

On Tue, 16 Oct 2012 12:23:24 -0400
Greg Troxel <gdt-2FjktZCtrC/QT0dZR+AlfA at public.gmane.org> wrote:

> 
> Does your tool need to be tahoe specific?  Can it back up to any remote
> filesystem accessed through the VFS layer?  If no, why not, and could
> you remove that limitation?
> 

It doesn't have to be tahoe-specific as such, and in theory it can be
extended to facilitate such operation mode, but I haven't really
considered adding such functionality.

Main reason why is that such vfs-to-vfs backup tool (with optional
network tunnel in-between) already exists in far superior form.

Plain rsync seem to be sufficient in most cases - it can backup
everything, supports flexible filtering, efficient deduplication
(hardlinks) and delta-transfers with compression over network.

Missing (in rsync) functionality might be at-rest compression can (and
probably should) be solved on vfs-layer in the form of either support
in destination fs or a layer on top.
State-keeping between transfers (e.g. write-only remote with
deduplication) and deduplication in general I think is solved nicely by
tools like unison, csync2 or git-annex.

Advanced filesystems, such as btrfs, provide even more options.

Of (solvable) problems in implementing such thing, I see "encoding"
metadata storage (e.g. that the contents are compressed), but xattrs on
destination fs can probably be used.

Deduplication is probably a hardest problem to solve in vfs-to-vfs case
- there are lots of ways to do it, which all seem to be non-universal -
hardlinking, rdiff, copying only new files, cow filesystems and their
features, git-like by-content addressing, etc.

I tend to think that most vfs-to-vfs operation requirements might
be addressed (if weren't already) by a much different tool (or maybe
extension to an existing one), which won't have to deal with http
requests, caching of the capability strings, metadata serialization and
the other complexity added by the fs-to-tahoe translation, but will
probably have a lot of tahoe-unrelated features (such as specific
deduplication mode) of it's own.

So while I see that it's possible to allow such use-case, I'm quite
unconvinced that it's worth the added complexity.

But I wonder, what particular usage do you have in mind that all the
existing tools seem to fail to address?


-- 
Mike Kazantsev // fraggod.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20121017/97f8b294/attachment.pgp>