Magic Folder on Linux: handle case when directory is removed from monitored directory

[dawuud]

We need to handle the case where a directory is removed or moved out of the monitored local directory.

Specifically this case needs to be specially handled only when the sub-directory was moved into the drop uploader directory and a inotify watch for that sub-directory was added. Further investigation is needed to determine the proper way to remove this inotify watch.

So far it seems that we will respond to either the IN_DELETE_SELF or IN_MOVE_SELF events for each of these sub-directory inotify "watches".

[daira]

It seems as though the IN_EXCL_UNLINK flag would help with this:

   IN_EXCL_UNLINK (since Linux 2.6.36)
          By default, when watching events on the children of a
          directory, events are generated for children even after
          they have been unlinked from the directory.  This can
          result in large numbers of uninteresting events for some
          applications (e.g., if watching /tmp, in which many
          applications create temporary files whose names are
          immediately unlinked).  Specifying IN_EXCL_UNLINK changes
          the default behavior, so that events are not generated for
          children after they have been unlinked from the watched
          directory.

[daira]

The constant for IN_EXCL_UNLINK isn't defined by Twisted (​https://twistedmatrix.com/trac/ticket/7789) but that's okay, we can define it ourselves, since Twisted passes through unknown flags.

[daira]

Note that this is not an issue on Windows, because ReadDirectoryChangesW only reads events that occur for paths that are still under the recursively-watched directory.

[dawuud]

Furthermore these additional tests should also demonstrate that a directory which has been moved into the drop upload directory will have the IN_EXCL_UNLINK behavior just like the upload directory itself.

[daira]

Add magic-folder keyword to all drop-upload tickets.

[dawuud]

I'm a little tiny bit frustrated trying to debug the unit tests. They seem to be written correctly but are producing unexpected results. Perhaps we've overlooked a simple and silly mistake somewhere.

I've pushed a commit to my dev branch which demonstrates some broken behavior; a directory with a file (small_tree) is moved into the upload directory. The directory and file are uploaded... however when an additional file is written to the subdir... it is not uploaded. So confusing!

​https://github.com/david415/tahoe-lafs/tree/2427.remove-uploader-watch.1

[daira]

Remember that we removed the autoAdd=True argument from the initial call to watch, in ​https://github.com/david415/tahoe-lafs/commit/e57645a9f15a7223eb33984cdd495a55b246e474. We did this because we believed that adding the extra explicit call to watch in _add_dir (​here) was the correct way to handle directories being moved into the monitored directory tree, since we had found that autoAdd does not implicitly add a watch in that case.

It's likely that the problem is in that code; I would suggest adding a print to make sure that watch is being called for the moved-in directory, and with the correct arguments.

[dawuud]

yess! i got the real tests passing in my dev branch but i'll have to fix the mock tests later tomorrow.

[daira]

\o/. Looking forward to seeing what the problem was.

[dawuud]

the problem was that i accidentally had some important code bits commented out... and the original problem was solved by our not using the autoadd feature. fixed here: ​https://github.com/david415/tahoe-lafs/commits/2427.remove-uploader-watch.1 ​https://github.com/david415/tahoe-lafs/commit/fe797eb76b88d3e31da842551b2abe2d5084b6e6

[daira]

Fixed on the 2438.otf-objective-4.2 branch.

[warner]

Milestone renamed