add 'tahoe mirror' command, use backupdb

[warner]

It would be nice to have a CLI tool which does a minimum-effort copy, from local disk into a Tahoe directory. This tool should behave like "rsync -a --delete": when done, the target directory should look exactly like the local disk's directory. By running the tool on a periodic basis, the Tahoe directory will contain a single most-recent-version backup of the local disk.

tahoe mirror ~/music tahoe:my-music

To make this as fast as possible, the default mode will use two assumptions:

• the target Tahoe directory has not been changed by other parties since the last invocation of 'tahoe mirror'
• any file changes will modify either their timestamp or filesize

Both assumptions can be disabled with argv flags, at the expense of doing more work. If both assumptions are accepted, then a null backup should require no network traffic and no file reads (only directory reads).

This command would use the 'backupdb', stored in ~/.tahoe/private/backupdb, as discussed in this thread: ​http://allmydata.org/pipermail/tahoe-dev/2008-May/000620.html . The backupdb would allow the client to quickly determine which files and directories have already been copied.

The proposed "tahoe backup" command (#598) would use the same backupdb.

[warner]

so, it looks like #598 ("tahoe backup": versioned shared backups) is more interesting right now, so I'll be working on it instead of "tahoe sync". To checkpoint my work so far: here's my pseudocode, and the stub of the CLI code.

create target directory

loop(localdir, target tahoe dir):
 fetch targetdir
 delete any:
  children that don't exist locally
  files that should be dirs
  dirs that should be files
 create missing dirs
 for each file in localdir:
  chk = upload(file) # uses backupdb to short-circuit
  if chk,metadata == targetdir[child]:
   continue
  else:
   set_child(child, chk+metadata)
 for each subdir in localdir:
  loop()

assuming upload() uses a backupdb successfully, a null backup with this
algorithm will read all targetdirs but will not upload or modify anything.

[warner]

starting point, just a stub of the CLI command

[warner]

after some discussion, we decided that "tahoe mirror" was a better name for this than "tahoe sync". "sync" implies bidirectionality, whereas "mirror" has a definite real-world side and looking-glass-world side.

So this ticket is about "tahoe mirror", which does whatever is necessary to make a target directory look like a source directory, without modifying the source directory. The "tahoe sync" idea (which makes the directories look the same, but is allowed to modify *both* directories) has been moved to #601.

[warner]

oops, forgot to modify the ticket description

[stockrt]

Warner, isn't this ticket about the functionality already provided by 'tahoe backup' #598?

Wouldn't be good to close this one?

[warner]

stockrt: nope, "tahoe backup" is defined to create successive timestamped snapshots, whereas "tahoe mirror" is defined to create/modify a single snapshot.

After you've used "tahoe backup ... alias:Backups" daily for a few days, you'll have:

• Backups/Latest/?...
• Backups/Archives?/2009-03-10/...
• Backups/Archives?/2009-03-11/...
• Backups/Archives?/2009-03-12/...

After you've used "tahoe mirror ... alias:Backups" daily for a few days (or a month, or just once), you'll have:

• Backups/...

If you used "tahoe backup ... alias:Backups" and then ignored the Backups/Archives/? directory, you'd get the same thing as you'd get with "tahoe mirror ... alias:Backups/Latest". But someone who wants just the latest copy would 1) be annoyed by the old archives piling up and 2) would be annoyed by the extra "Latest/" subdirectory that they didn't ask for. That's why it seems like a separate command would be useful. (but, not as useful as "tahoe backup", which is why "tahoe mirror" got de-prioritized).