Magic Folder on Mac OS X and other BSDs

[davidsarah]

The Magic Folder feature currently only works on Linux (where it uses inotify to detect filesystem changes), and Windows (where it uses ReadDirectoryChangesW). This ticket is about supporting the same feature on Mac OS X and possibly other BSD-based operating systems.

Mac OS X has the ​fsevents, and all BSDs including Mac OS X have kqueue/kevent. If I understand correctly, neither fsevents nor kqueue/kevent give notifications of which files have changed, so the implementation will have to scan the directory and look at last-modified times to determine that. (This is occasionally also necessary for correctness on Linux and Windows; see #1430.)

[davidsarah]

Python's ​select module supports kqueue, which is a lot easier and less error-prone than accessing it via ctypes or an extension module. Note that a ​critical bug in that support was fixed in Python 2.6.5. (2.7 was released after 2.6.5 and has this fix.)

[malaparte]

I thought I'd just save this thought here, because I don't know if I have time to look at it closely. Have you considered 'watchdog' ​http://packages.python.org/watchdog/ to watch for changes on a folder across platforms?

[davidsarah]

watchdog uses a C ​extension module to access the fsevents API on OS X. I would strongly prefer to use ctypes for this (if needed, i.e. if we don't use kqueue instead) and not add any more native code dependencies.

[davidsarah]

Also,

davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet watchdog

http://cloc.sourceforge.net v 1.09  T=4.0 s (15.0 files/s, 2563.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          36           1286           2238           4010
C                                3            168            473            738
Bourne Shell                    13             71            188            273
make                             3             46              5            193
DOS Batch                        2             27              1            172
XML                              1              0              0            155
C/C++ Header                     1             28             68             93
YAML                             1              0              0             21
-------------------------------------------------------------------------------
SUM:                            60           1626           2973           5655
-------------------------------------------------------------------------------

plus watchdog's ​PyYaml, argh and pathtools dependencies

compared to:

davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-x86_64.egg/twisted/internet/inotify.py

http://cloc.sourceforge.net v 1.09  T=0.5 s (2.0 files/s, 810.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             84            140            181
-------------------------------------------------------------------------------
davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/src/allmydata/frontends/drop_upload.py

http://cloc.sourceforge.net v 1.09  T=0.5 s (2.0 files/s, 246.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             25             10             88
-------------------------------------------------------------------------------

plus how ever much code it takes to support Windows (about ​200-300 code lines), and BSD/OS X.

[davidsarah]

I should mention that I don't care about lines of code per se; I'm just using it as a rough proxy measure for complexity and opportunity-for-bugs.

[daira]

Add magic-folder keyword to all drop-upload tickets.