magic-folders synchronization diagrams

[Brian Warner]

In the Nuts+Bolts videochat this morning, we talked a lot about how the
"Magic Folders" feature (currently under development by the fine folks
at LAE) should work, in particular how you detect and resolve conflicts.

I'll leave the summary of that discussion to others, but I wanted to
post the two diagrams we wrote up.

In both cases, for each "magic folder" being synchronized between
multiple computers, there's a tahoe-side object for each computer (i.e.
each uncoordinated writer). This ensures that no tahoe mutable object is
written to by more than one party, thus obeying Tahoe's "Prime
Coordination Directive": Don't Do That.

In the first diagram, the magic folder's entire structure is encoded
into a single tahoe mutable directory. In this mutable directory, the
keys contain delimiter-joined pathnames (maybe "/", maybe something
else), and the values contain filecaps (never subdirs). We're
provisionally calling this sort of thing a "DMT", for Distributed
Mutable Tree. This reduces the number of tahoe objects that must be
followed by the other participants down to one (vs them needing to poll
a whole mess of subdirectories), and provides clear snapshots of
published contents.

In the second diagram, each node uses something like "tahoe backup" to
produce a series of snapshots of the local magic-folder directory
structure. These snapshots include a "metadata" directory as a sibling
of the actual contents, and this metadata includes a list of which
versions the node saw the *other* nodes publishing. For example, Alice's
backups will include a note that says "when I made this snapshot, I
observed Bob's current snapshot to be version/seqnum XYZ". Bob can then
use this information to decide whether Alice was aware of one of his
changes or not, and thus distinguish between the "this is a regular
changed file" and "this is a simultaneous-write conflict" cases. Daira
is going to write up an algorithm for identifying these cases.

In a way, this is just the latest iteration of a conversation we've been
having for years, about how to chose a synchronization protocol that
isn't too hard to build and works "good enough" in the most common
cases, but also tries to avoid data loss in the bad cases. But now that
magic folders are actually happening, it's time to pick something and
run with it :).

cheers,
 -Brian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sync-1.png
Type: image/png
Size: 74760 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20150505/3017ce7a/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sync-2.png
Type: image/png
Size: 99551 bytes
Desc: not available
URL: <http://tahoe-lafs.org/pipermail/tahoe-dev/attachments/20150505/3017ce7a/attachment-0003.png>