drop-upload on Mac OS X and other BSDs

[davidsarah]

The drop-upload feature (#1429) uses inotify to detect filesystem changes, and therefore works only on Linux. This ticket is about supporting the same feature on Mac OS X and possibly other BSD-based operating systems. Mac OS X has the ​fsevents, and all BSDs including Mac OS X have kqueue/kevent. If I understand correctly, neither fsevents nor kqueue/kevent give notifications of which files have changed, so the implementation will have to scan the directory and look at last-modified times to determine that. (This is occasionally also necessary for correctness on Linux; see #1430.)

[davidsarah]

Python's ​select module supports kqueue, which is a lot easier and less error-prone than accessing it via ctypes or an extension module. Note that a ​critical bug in that support was fixed in Python 2.6.5. (2.7 was released after 2.6.5 and has this fix.)

[malaparte]

I thought I'd just save this thought here, because I don't know if I have time to look at it closely. Have you considered 'watchdog' ​http://packages.python.org/watchdog/ to watch for changes on a folder across platforms?

[davidsarah]

watchdog uses a C ​extension module to access the fsevents API on OS X. I would strongly prefer to use ctypes for this (if needed, i.e. if we don't use kqueue instead) and not add any more native code dependencies.

[davidsarah]

Also,

davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet watchdog

http://cloc.sourceforge.net v 1.09  T=4.0 s (15.0 files/s, 2563.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          36           1286           2238           4010
C                                3            168            473            738
Bourne Shell                    13             71            188            273
make                             3             46              5            193
DOS Batch                        2             27              1            172
XML                              1              0              0            155
C/C++ Header                     1             28             68             93
YAML                             1              0              0             21
-------------------------------------------------------------------------------
SUM:                            60           1626           2973           5655
-------------------------------------------------------------------------------

plus watchdog's ​PyYaml, argh and pathtools dependencies

compared to:

davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-x86_64.egg/twisted/internet/inotify.py

http://cloc.sourceforge.net v 1.09  T=0.5 s (2.0 files/s, 810.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             84            140            181
-------------------------------------------------------------------------------
davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/src/allmydata/frontends/drop_upload.py

http://cloc.sourceforge.net v 1.09  T=0.5 s (2.0 files/s, 246.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             25             10             88
-------------------------------------------------------------------------------

plus how ever much code it takes to support Windows (about ​200-300 code lines), and BSD/OS X.

[davidsarah]

I should mention that I don't care about lines of code per se; I'm just using it as a rough proxy measure for complexity and opportunity-for-bugs.