[tahoe-dev] Keeping local file system and Tahoe store in sync

[Shawn Willden]

On Friday 06 February 2009 04:45:38 pm Brian Warner wrote:
> For linux, look at 'dnotify', I think it's closer to what you want.

As Jeremy said, inotify is much lighter-weight and more scalable than dnotify, 
but still not ideal.  The problem with inotify is that you have to register 
for every directory you want to watch, and each watch eats up kernel memory.  
I'm not sure if you can use it effectively to monitor a whole file system, 
and even if you can it's painful and heavy compared to fsevents.

> If I remember right, for OS-X, fsevents should give you reliable "this
> directory might have been modified" information (i.e. it won't give you a
> false negative, but it might give you a false positive). You must scan
> every directory it tells you about, but sometimes nothing will have
> changed.

Right, which is beautiful for this purpose.  Not surprising, since FSEvents 
was created precisely to facilitate Apple's backup solution.

> > [2] The files uploaded may be full revisions or diffs. In either case,
> > the encryption key used will be the full revision hash.
>
> Be careful to not use the same key for multiple purposes. Tahoe always uses
> a tagged hash when deriving a key in this fashion:

I don't see a security issue for reusing keys in this case, but I just 
remembered a really important reason -- storage IDs have to be unique and 
they're a hash of the encryption key.

Okay.  So I'll derive signature and diff keys from the content hash, rather 
than using the same key for everything.  That's easy enough, and still allows 
me to use the content hash as the key in the read cap files.

>  key = SHA256d(netstring("file contents to encryption key v1") + contents)

Yes, hash tagging is a good idea, for all sorts of reasons.

> (I'll try to review and respond to the rest of your note later)

I appreciate it!

Thanks,

	Shawn.