#1432 closed enhancement (fixed)

Magic Folder on Mac OS X and other BSDs

Reported by: davidsarah Owned by: GitHub <noreply@…>
Priority: normal Milestone: undecided
Component: code-frontend-magic-folder Version: 1.8.2
Keywords: fsevents mac bsd magic-folder Cc:
Launchpad Bug:

Description (last modified by daira)

The Magic Folder feature currently only works on Linux (where it uses inotify to detect filesystem changes), and Windows (where it uses ReadDirectoryChangesW). This ticket is about supporting the same feature on Mac OS X and possibly other BSD-based operating systems.

Mac OS X has the fsevents API, and all BSDs including Mac OS X have kqueue/kevent. If I understand correctly, neither fsevents nor kqueue/kevent give notifications of which files have changed, so the implementation will have to scan the directory and look at last-modified times to determine that. (This is occasionally also necessary for correctness on Linux and Windows; see #1430.)

Change History (52)

comment:1 Changed at 2011-07-17T02:21:50Z by davidsarah

  • Description modified (diff)
  • Keywords bsd added
  • Summary changed from drop-upload on Mac OS X to drop-upload on Mac OS X and other BSDs

comment:2 Changed at 2011-07-25T12:26:43Z by davidsarah

  • Type changed from defect to enhancement

comment:3 Changed at 2011-08-15T19:38:42Z by davidsarah

Python's select module supports kqueue, which is a lot easier and less error-prone than accessing it via ctypes or an extension module. Note that a critical bug in that support was fixed in Python 2.6.5. (2.7 was released after 2.6.5 and has this fix.)

comment:4 Changed at 2012-03-29T19:51:21Z by davidsarah

  • Priority changed from major to normal

comment:5 follow-up: Changed at 2012-11-15T06:07:59Z by malaparte

I thought I'd just save this thought here, because I don't know if I have time to look at it closely. Have you considered 'watchdog' http://packages.python.org/watchdog/ to watch for changes on a folder across platforms?

comment:6 in reply to: ↑ 5 Changed at 2012-11-15T06:11:24Z by malaparte

Last edited at 2012-11-15T06:12:02Z by malaparte (previous) (diff)

comment:7 Changed at 2012-11-15T23:39:14Z by davidsarah

watchdog uses a C extension module to access the fsevents API on OS X. I would strongly prefer to use ctypes for this (if needed, i.e. if we don't use kqueue instead) and not add any more native code dependencies.

Last edited at 2012-11-16T00:31:36Z by davidsarah (previous) (diff)

comment:8 Changed at 2012-11-16T00:22:50Z by davidsarah

Also,

davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet watchdog

http://cloc.sourceforge.net v 1.09  T=4.0 s (15.0 files/s, 2563.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          36           1286           2238           4010
C                                3            168            473            738
Bourne Shell                    13             71            188            273
make                             3             46              5            193
DOS Batch                        2             27              1            172
XML                              1              0              0            155
C/C++ Header                     1             28             68             93
YAML                             1              0              0             21
-------------------------------------------------------------------------------
SUM:                            60           1626           2973           5655
-------------------------------------------------------------------------------

plus watchdog's PyYaml, argh and pathtools dependencies

compared to:

davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/support/lib/python2.6/site-packages/Twisted-12.0.0-py2.6-linux-x86_64.egg/twisted/internet/inotify.py

http://cloc.sourceforge.net v 1.09  T=0.5 s (2.0 files/s, 810.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             84            140            181
-------------------------------------------------------------------------------
davidsarah@shinier:~/tahoe/git$ cloc --no3 --quiet trunk/src/allmydata/frontends/drop_upload.py

http://cloc.sourceforge.net v 1.09  T=0.5 s (2.0 files/s, 246.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             25             10             88
-------------------------------------------------------------------------------

plus how ever much code it takes to support Windows (about 200-300 code lines), and BSD/OS X.

comment:9 Changed at 2012-11-16T00:29:27Z by davidsarah

I should mention that I don't care about lines of code per se; I'm just using it as a rough proxy measure for complexity and opportunity-for-bugs.

Last edited at 2015-10-28T23:13:52Z by daira (previous) (diff)

comment:10 Changed at 2014-12-02T19:48:43Z by warner

  • Component changed from code-frontend to code-frontend-drop-upload
  • Description modified (diff)

comment:11 Changed at 2015-06-01T16:11:09Z by daira

  • Keywords magic-folder added

Add magic-folder keyword to all drop-upload tickets.

comment:12 Changed at 2015-10-28T23:18:23Z by daira

  • Description modified (diff)
  • Keywords drop-upload removed
  • Summary changed from drop-upload on Mac OS X and other BSDs to Magic Folder on Mac OS X and other BSDs

comment:13 Changed at 2015-10-28T23:19:16Z by daira

  • Description modified (diff)

comment:14 Changed at 2015-12-04T21:14:33Z by daira

SpiderOak has an fsevents-based OS X filesystem watcher which is open-source (GPL): https://github.com/SpiderOak/spideroak_osx_fsevents

[Edit: I did not mean that we should necessarily use this as-is.]

Last edited at 2016-04-05T15:35:52Z by daira (previous) (diff)

comment:15 Changed at 2016-01-27T16:09:39Z by dawuud

step number one seems to be: find an OSX VPS provider!

comment:16 Changed at 2016-04-07T14:03:34Z by zooko

I think we should start with https://github.com/SpiderOak/spideroak_osx_fsevents and then consider changing it or switching to other techniques or whatever after we have it running.

comment:17 Changed at 2016-04-07T16:50:18Z by zooko

Okay I looked at https://github.com/SpiderOak/spideroak_osx_fsevents/blob/master/main.c and other code in watchdog and so on and I changed my mind, because https://github.com/SpiderOak/spideroak_osx_fsevents doesn't have a Python API.

We already have a ctypes-based API to Linux and Windows, so I guess our code would be more consistent, and easy for us to write, if we also wrote a ctypes-based API to OSX?

comment:18 Changed at 2016-04-13T12:03:40Z by dawuud

Of the several steps needed to setup the Apple async filesystem events API we need to firstly call FSEventStreamCreate.

There's a snippet of code showing the FSEventStreamCreate being used:

https://developer.apple.com/library/mac/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html

It doesn't look like ctypes allows the creation of c structs or objective c objects unless you redefine them and create them the ctypes way. So initially I was thinking that maybe recreating all the necessary structs/objective-c-objects in python ctypes manually is kinda tricky... and requires us to have the Apple source code or at least the .h header files with the struct/object definitions.

Perhaps the pyobjc library can help us to compose these objective-c structures so that we can use ctypes to pass them into API functions. Here need a python equivalent for composing the CFArray of CFString which holds the filesystems paths to watch:

    CFStringRef mypath = CFSTR("/path/to/scan");

    CFArrayRef pathsToWatch = CFArrayCreate(NULL, (const void **)&mypath, 1, NULL);

    void *callbackInfo = NULL; // could put stream-specific data here.

    FSEventStreamRef stream;

    CFAbsoluteTime latency = 3.0; /* Latency in seconds */

The pyobjc module requies xcode and when I pip installed it after properly installed xcode it downloaded many more modules; I wonder if this thing is too heavy weight. We probably want to avoid heavyweight dependencies?

I also discovered that google wrote a ctypes wrapper for Apple objective-c structures in this project:

https://github.com/google/grr https://github.com/google/grr/blob/master/grr/client/osx/objc.py

oh and i also found this other ctypes wrapper API here: https://github.com/wbond/oscrypto/blob/master/oscrypto/_osx/_core_foundation_ctypes.py

Last edited at 2016-04-13T12:20:09Z by dawuud (previous) (diff)

comment:19 Changed at 2016-04-14T14:24:33Z by dawuud

Currently I have a failed attempt at creating this CFArray of CFString`s; running it results in a segmentation fault... however the VM i'm using won't write a core file. What to do?

(virtenv-objc) xcloud114:~ Xcloud$ python test.py
Segmentation fault: 11
#!/usr/bin/env python                                                                                                             

import ctypes
from ctypes import POINTER, byref, create_string_buffer, addressof
from ctypes import cdll, c_void_p
from ctypes.util import find_library

# kCFStringEncodingUTF8                                                                                                           
UTF8 = 134217984
# kCFAllocatorDefault                                                                                                             
CF_DEFAULT_ALLOCATOR = None

core_services = cdll.LoadLibrary(find_library('CoreServices'))
core_foundation = cdll.LoadLibrary(find_library('CoreFoundation'))

def PyStringToCFString(pystring):
    return core_foundation.CFStringCreateWithCString(CF_DEFAULT_ALLOCATOR,
                                  pystring.encode('utf8'),
                                  UTF8)

my_cfarray = core_foundation.CFArrayCreate(None, None, 0, ctypes.c_int.in_dll(core_services, "kCFTypeArrayCallBacks"))
my_cfstr = PyStringToCFString("/Users/Xcloud/magic-folder")

core_foundation.CFArrayAppendValue(my_cfarray, my_cfstr)

comment:20 Changed at 2016-04-15T12:49:10Z by dawuud

today i wrote a partial working prototype for Mac OS X using the python watchdog library, here in my dev branch:

https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog.0

many of the magic-folder tests pass. however it seems that test_magic_folder and test_move_tree fail due to watchdog not recursively watching directories or not adding a watch to newly added directories or some such thing.

anyway we should be able to replace this usage of the watchdog api with a ctypes equivalent for OSX if we want to do that.

Last edited at 2016-04-15T12:50:00Z by dawuud (previous) (diff)

comment:21 Changed at 2016-04-15T12:59:01Z by zooko

Whoo-hoo! Way to go!

comment:22 Changed at 2016-04-16T15:46:29Z by dawuud

this actually is a BSD compatibility change since getting magic-folder to work with this watchdog based inotify API actually makes it work on all the BSD operating systems using kqueue and it should also work on Mac OS X using the native apple filesystem events api.

we could of course replace it with two ctypes based inotify APIs. one for kqueue and one for Mac OS X fs events api...

currently there are only two or three unit tests that fail and these failures are not so terrible... it fails because it's emitting more events than the unit tests expect.

comment:23 Changed at 2016-05-02T13:56:31Z by dawuud

the latest watchdog inotify magic-folder branch is here: https://github.com/tahoe-lafs/tahoe-lafs/tree/1432.osx-watchdog.0

you may note that the travis ci tests fail but this is due to changes in the dot travis yaml file as we attempt to get it working in a linux container and an osx container. this difficulty with the travis configuration was due to the removal of the config options specifying python and instead telling travis to run in either a linux or osx container. then we are required to manually install many package dependencies so that pip and tahoe-lafs function properly and so on. i have previously gotten the travis configuration file to work properly for linux and osx... but this indeed involved changing other configuration options such as enabling sudo.

what's the next step to getting this merged into tahoe master or magic-folder stable?

comment:24 Changed at 2016-05-02T14:59:20Z by daira

dawuud: please paste the current test failures.

comment:25 Changed at 2016-05-03T15:33:25Z by dawuud

[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1104, in <lambda>
    d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 6 != 4

allmydata.test.test_magic_folder.RealTest.test_move_tree
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 687, in <lambda>
    d.addCallback(lambda ign: self._check_uploader_count('objects_not_uploaded', 1, magic=self.bob_magicfolder))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 211, in _check_uploader_count
    expected)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 2 != 1

allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_bob
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 169, in _check_file
    os.unlink(path_u)
exceptions.OSError: [Errno 2] No such file or directory: '/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/MockTest/test_magic_fo
lder/gIRrj8/temp/local_dir/tempfile'

allmydata.test.test_magic_folder.MockTest.test_magic_folder
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 169, in _check_file
    os.unlink(path_u)
exceptions.OSError: [Errno 2] No such file or directory: '/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_fo
lder/ff93bG/temp/local_dir/tempfile'

allmydata.test.test_magic_folder.RealTest.test_magic_folder
-------------------------------------------------------------------------------
Ran 35 tests in 171.881s

FAILED (failures=2, errors=2, successes=31)

comment:26 Changed at 2016-05-09T14:54:06Z by dawuud

the above test results were actually from an earlier commit; this commit id 8939fbf16baa2cbd9ede7b23037b61dfbe323cdd

seems to have fewer test errors and failures than the HEAD of that branch. This is of course because the code review "corrections" caused more tests to fail... I'll have to track down which changes cause the problems. I suspect they are unicode related changes.

Furthermore our inotify/watchdog API must implement two more features:

  1. allow the user to set an events mask to select which events to get notifications for
  1. event attributes. the event messages may contain information about the file object which caused the events such as: file type (e.g. file or directory).

HOWEVER, given that the watchdog API passes nearly all the magic-folder unit tests, this means either the watchdog api is good enough or our unit test coverage sucks.

comment:27 Changed at 2016-05-24T12:24:13Z by dawuud

i made a new dev branch with the watchdogi inotify stable changes on top of the latest stable magic-folder branch (stable 13): https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.1

pull request here: https://github.com/tahoe-lafs/tahoe-lafs/pull/287

comment:28 Changed at 2016-05-25T14:36:49Z by dawuud

new dev branch based off of meejah's magic-folder stable 18 branch here: https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.3

no errors, two failures:

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 183, in _check_file
    previously_disappeared + 1)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 0 != 1

allmydata.test.test_magic_folder.RealTest.test_magic_folder
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1131, in <lambda>
    d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 6 != 4

allmydata.test.test_magic_folder.RealTest.test_move_tree
-------------------------------------------------------------------------------
Ran 35 tests in 182.773s

FAILED (failures=2, successes=33)

comment:29 Changed at 2016-05-25T15:29:30Z by dawuud

however a simple change, ignore events outside the watched directory shows the score a bit closer to the target:

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 183, in _check_file
    previously_disappeared + 1)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in 
assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in 
_baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 0 != 1

allmydata.test.test_magic_folder.RealTest.test_magic_folder
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 1131, in <lambda>
    d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4))
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in 
assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in 
_baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 3 != 4

allmydata.test.test_magic_folder.RealTest.test_move_tree
-------------------------------------------------------------------------------
Ran 35 tests in 188.915s

FAILED (failures=2, successes=33)

comment:30 Changed at 2016-05-26T14:29:05Z by dawuud

new error output from tests show all tests pass except one:

    test_magic_folder ... set pending delay                                                                                                   [2/1850]
init INotifyEventHandler
START READING BEGIN
START READING END
PROCESS EVENT <FileCreatedEvent: src_path='/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_folder/Q5TLUN/te$p/local_dir/short'>
PROCESS EVENT <DirModifiedEvent: src_path='/Users/Xcloud/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_magic_folder/Q5TLUN/te$
p/local_dir'>
DO CALLBACKS
                                               [FAIL]
stopReading begin
stopReading end
wait until stopped

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/test_magic_folder.py", line 188, in _check_file
    self.failUnlessReallyEqual(actual_data, data)
  File "/Users/Xcloud/tahoe-lafs/src/allmydata/test/common_util.py", line 35, in failUnlessReallyEqual
    self.failUnlessEqual(a, b, msg=msg)
  File "/Users/Xcloud/virtenv-tahoe/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 'test' != 'different'

allmydata.test.test_magic_folder.RealTest.test_magic_folder
-------------------------------------------------------------------------------
Ran 2 tests in 5.686s

FAILED (failures=1, successes=1)

comment:31 Changed at 2016-05-30T15:26:12Z by dawuud

in another dev branch i saved a commit with added debug print statements so i could track down the root cause of the above unit test failure: https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.3.wip.1

what is happening is that this test case fails: https://github.com/david415/tahoe-lafs/blob/e451e2eaa410d84aa3ae0b62a5aaf5a4d4aa786b/src/allmydata/test/test_magic_folder.py#L1240

_check_file is failing on the temporary file case; specifically here's the comparison that fails:

https://github.com/david415/tahoe-lafs/blob/e451e2eaa410d84aa3ae0b62a5aaf5a4d4aa786b/src/allmydata/test/test_magic_folder.py#L182

yield self.failUnlessReallyEqual(self._get_count('uploader.objects_disappeared'),
                                             previously_disappeared + 1)
uploader.objects_disappeared 0
previously_disappeared + 1 == 1

OK so it uploads the temporary file when it shouldn't. But is that really so bad? Seems better than not uploading enough.

comment:32 Changed at 2016-06-01T07:33:51Z by dawuud

I have had two conversations about magic-folder unit test race conditions; one with Daira and one with Meejah. Currently the plan is for Meejah to fix/refactor all the uploader unit tests after figuring out a better API for uploader testing similar in spirit to how we refactored the downloader tests. Here's the ticket for that:

https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2790

Last edited at 2016-06-07T15:54:32Z by dawuud (previous) (diff)

comment:33 Changed at 2016-06-22T15:59:41Z by dawuud

here i've put the watchdog inotify changes on top of meejah's stable 18 branch: https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.4

comment:34 Changed at 2016-06-22T16:10:16Z by dawuud

oh dear the linux build passes but the osx build fails with the current travis yaml configuration::

RuntimeError?: You are linking against OpenSSL 0.9.8, which is no longer support by the OpenSSL project. You need to upgrade to a newer version of OpenSSL.

there's a ticket regarding automated testing on OSX here: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2789

comment:35 Changed at 2016-08-16T17:55:57Z by warner

BTW, that openssl thing was fixed. The travis OS-X builder was using the wrong version of python, and failing to pick up the newer openssl library. If you update the PR, travis should run with the proper version.

comment:37 Changed at 2016-08-29T13:31:04Z by dawuud

currently on the darwin platform fails at least this test: allmydata.test.test_magic_folder.RealTest?.test_delete

I don't understand why this fails. I added a print statement to display the platform filesystem encoding which is UTF-8

` test_delete ... set pending delay

init INotifyEventHandler

START READING BEGIN

START READING END

PROCESS EVENT <FileCreatedEvent: src_path='/Users/travis/build/tahoe-lafs/tahoe-lafs/_trial_temp/allmydata.test.test_magic_folder/RealTest/test_delete/nvdp3V/temp/local_dir/foo'>

FILESYSTEM ENCODING: utf-8

Exception in thread Thread-8:

Traceback (most recent call last):

File "/System/Library/Frameworks/Python?.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in bootstrap_inner

self.run()

File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/observers/api.py", line 199, in run

self.dispatch_events(self.event_queue, self.timeout)

File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/observers/api.py", line 368, in dispatch_events

handler.dispatch(event)

File "/Users/travis/build/tahoe-lafs/tahoe-lafs/.tox/py27/lib/python2.7/site-packages/watchdog/events.py", line 322, in dispatch

self.on_any_event(event)

File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/watchdog/inotify.py", line 73, in on_any_event

self.process(event)

File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/watchdog/inotify.py", line 47, in process

event_filepath_u = abspath_expanduser_unicode(event_filepath_u, base=self._path)

File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/util/fileutil.py", line 310, in abspath_expanduser_unicode

precondition_abspath(base)

File "/Users/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/util/fileutil.py", line 276, in precondition_abspath

raise AssertionError?("an abspath must be a Unicode string")

AssertionError?: an abspath must be a Unicode string `

comment:38 Changed at 2016-11-21T21:28:32Z by exarkun

Some changes built on david's last branch - https://github.com/exarkun/tahoe-lafs/tree/1432.osx-watchdog-stable.9

In particular, this adds a direct inotify-level test suite (copied from Twisted!) and makes some of the tests pass - including the IN_CLOSE_WRITE notification.

comment:39 Changed at 2016-11-23T20:49:22Z by dawuud

more progress here in this dev branch. i've changed the inotify test suite to skip many of the tests for inotify features that we do not use or need. I also modified some of the tests so that they now pass when using either the watchdog inotify and the linux native inotify. not yet tested with our windows inotify implementation.

https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.9

comment:40 Changed at 2016-11-23T22:45:46Z by dawuud

now that we have a inotify test suite perhaps we can remove the "real" inotify magic folder tests?

in the above mentioned dev branch i've removed the real tests in this commit here: https://github.com/david415/tahoe-lafs/commit/451dcf093b26b475378ef588e2cf68b40697cc8c

comment:41 Changed at 2016-11-25T00:32:38Z by meejah

Is the reactor.callFromThread() because the "watchdog" library is running the on_any_event from its own thread(s)?

Otherwise looks plausible. And yes I'd like to get rid of some of the "long" magic-folder tests especially. If our test show that all 3 underlying inotify-like implementations are doing the right thing, then yes it would be great to get rid of the "Real" vs "Mock" dual tests in magic-folder as well! (i.e. just have the Mock ones).

Version 0, edited at 2016-11-25T00:32:38Z by meejah (next)

comment:42 Changed at 2016-12-08T19:37:30Z by dawuud

this latest dev branch is currently in a pull-request https://github.com/tahoe-lafs/tahoe-lafs/pull/381

all tests pass except for the appveyor testing on windows. currently i can see that the inotify tests are failing for windows; the test_attrib fails and test_closeWrite timesout.

meejah and i plan to look at this soon.

comment:43 Changed at 2017-01-09T16:02:08Z by dawuud

i'm trying to make this branch pass all the tests. in my latest commit i'm trying to make the test_inotify tests be skipped on windows... we can make the tests pass on windows later. for now i'd like to get the bsd changes merged if possible.

comment:44 Changed at 2017-02-14T23:53:08Z by dawuud

my latest dev branch: https://github.com/david415/tahoe-lafs/tree/1432.osx-watchdog-stable.10

all tests pass. the magic-folder "real inotify" tests have been disabled. the watch inotify implementation has it's own test suite that was borrowed and inspired from twisted's inotify test suite.

Last edited at 2017-02-15T18:16:05Z by dawuud (previous) (diff)

comment:45 Changed at 2017-02-15T18:13:55Z by dawuud

Last edited at 2017-02-15T18:14:28Z by dawuud (previous) (diff)

comment:46 Changed at 2017-02-20T18:31:25Z by dawuud

attempting to skip the inotify tests on windows: https://github.com/tahoe-lafs/tahoe-lafs/pull/401

the tests seem to be skipped but prints dirty reactor errors like this:

test_closeWrite ... [SKIPPED] Traceback (most recent call last): Failure: twisted.trial.util.DirtyReactorAggregateError?: Reactor was unclean. DelayedCalls?: (set twisted.internet.base.DelayedCall?.debug = True to debug) <DelayedCall? 0x4af3eb8 [0.00999999046326s] called=0 cancelled=0 LoopingCall?<0.01>(INotify._poll, *(<function <lambda> at 0x069DE770>, 1487535067.863), {})()> [ERROR]

why!? how to fix?

comment:47 Changed at 2017-02-20T20:50:29Z by dawuud

fixed it

comment:48 Changed at 2018-07-26T01:02:01Z by cypher

Freshly resurrected in PR 516.

comment:49 Changed at 2019-01-31T19:47:50Z by exarkun

Bad news. Watchdog upstream have decided to stop supporting Python 2.

https://github.com/gorakhargosh/watchdog/issues/516

comment:50 Changed at 2019-02-22T19:21:54Z by exarkun

They changed their mind. There are some other issues, though, like events being missed with the fsevents backend on macOS.

comment:51 Changed at 2019-03-14T17:02:13Z by GitHub <noreply@…>

  • Owner set to GitHub <noreply@…>
  • Resolution set to fixed
  • Status changed from new to closed

In c1e6f08/trunk:

Merge pull request #558 from tahoe-lafs/1432.watchdog-magic-folder-with-eliot

Add support for macOS to Magic-Folders.

Fixes: ticket:1432

comment:52 Changed at 2019-03-14T17:34:25Z by exarkun

Note to all, the above merge really did only add macOS support - not BSD. Sorry. Tickets should be about one thing, though.

Note: See TracTickets for help on using tickets.