[tahoe-lafs-trac-stream] [Tahoe-LAFS] #1433: drop-upload: also monitor subdirectories

[Tahoe-LAFS]

#1433: drop-upload: also monitor subdirectories
-------------------------------------+-------------------------------------
     Reporter:  davidsarah           |      Owner:
         Type:  enhancement          |     Status:  new
     Priority:  major                |  Milestone:  undecided
    Component:  code-frontend-drop-  |    Version:  1.8.2
  upload                             |   Keywords:  drop-upload usability
   Resolution:                       |  otf-magic-folder-objective2
Launchpad Bug:                       |
-------------------------------------+-------------------------------------

Comment (by daira):

 On 07/04/15 21:05, David Stainton wrote:>
 > a couple of my initial and trife observations:
 > 1.
 > Looks like the latest Twisted inotify class has got a recursive watch.
 > I'm not saying we need to use that version of Twisted; it's a simple
 loop we can implement.
 >
 https://twistedmatrix.com/trac/browser/tags/releases/twisted-15.0.0/twisted/internet/inotify.py#L350

 This has been there since the inotify support was added in Twisted 10.1.0,
 in fact.

 > 2.
 > I looked at the source code of the twisted INotify class to see if it's
 `watch` method could be called more than once and behave properly.
 > It looks like it does support this usage.

 Yes, it does, but the `recursive`/`autoAdd` support looks sufficient so we
 shouldn't need this. (We also don't need it for Windows because
 `ReadDirectoryChangesW` has a similar flag to watch a subdirectory tree.)

 > 3.
 > I am not clear on why drop_upload.py uses this inotify mask:
 >
 >   `mask = inotify.IN_CLOSE_WRITE | inotify.IN_MOVED_TO |
 inotify.IN_ONLYDIR`
 >
 > Combining all these definitions:
 > IN_MOVED_TO - File moved into watched directory
 > IN_CLOSE_WRITE - File opened for writing was closed
 > IN_ONLYDIR - Only watch pathname if it is a directory.
 >
 > This means we only watch subdirectories that have been created or moved
 into the directory. Did I understand things correctly?

 See comment:5 .

 > 4.
 > the inotify man page states:
 >
 >        If monitoring an entire directory subtree, and a new subdirectory
 is created in that tree, be aware that by the time you create a watch for
 the new subdirectory, new files may already have been created in the
 subdirectory.  Therefore, you may want to scan the contents of the
 subdirectory immediately after adding the watch.
 >
 > This obviously means we should construct a mask specifically to catch
 newly added directories like this:
 >
 >   `mask = inotify.IN_MOVED_TO | inotify.IN_ONLYDIR`
 >
 > and then add a specific inotify watch for that subdirectory... AND add
 recursively add that directory and its contents to the upload queue.

 The `autoAdd` flag already adds watches for children of newly added
 directories (and has done so since Twisted 10.1.0):

 https://twistedmatrix.com/trac/browser/tags/releases/twisted-10.1.0/twisted/internet/inotify.py#L259

 It also synthesizes `IN_CREATE` events for files and subdirectories found
 in those newly added directories (see the `_addChildren` method in
 `inotify.py`).

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1433#comment:6>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage