Magic Folder: tests fail on Travis-CI with UnicodeEncodeError from INotify.watch

[daira]

​https://travis-ci.org/tahoe-lafs/tahoe-lafs/jobs/65770833

allmydata.test.test_drop_upload.RealTest.test_drop_upload ... Traceback (most recent call last):
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/test/test_drop_upload.py", line 57, in _create_uploader
    dbfile, inotify=self.inotify, pending_delay=0.2)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/src/allmydata/frontends/drop_upload.py", line 75, in __init__
    recursive=True)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-15.2.1-py2.7-linux-x86_64.egg/twisted/internet/inotify.py", line 360, in watch
    recursive=False)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-15.2.1-py2.7-linux-x86_64.egg/twisted/internet/inotify.py", line 368, in watch
    return self._addWatch(path, mask, autoAdd, callbacks)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-15.2.1-py2.7-linux-x86_64.egg/twisted/internet/inotify.py", line 190, in _addWatch
    wd = self._inotify.add(self._fd, path.path, mask)
  File "/home/travis/build/tahoe-lafs/tahoe-lafs/support/lib/python2.7/site-packages/Twisted-15.2.1-py2.7-linux-x86_64.egg/twisted/python/_inotify.py", line 40, in add
    wd = libc.inotify_add_watch(fd, path, mask)
ctypes.ArgumentError: argument 2: <type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode character u'\u0101' in position 126: ordinal not in range(128)
[ERROR]

[daira]

This needs a call to NonASCIIPathMixin.unicode_or_fallback​ to construct the filename. It would also be good if we could get Travis to accept Unicode filenames, since that provides better test coverage.

[daira]

See #2268.

[daira]

Oh wait, I think this is a different error than I originally thought: it could be due to the FilePath's path attribute holding a Unicode string rather than a str.

This is strictly speaking a Twisted bug: in Twisted 15.2.1 which is the version used in this test, FilePath is supposed to accept Unicode paths (and so inotify.py should be using path._asBytesPath() instead of path.path ​here). Nevertheless, constructing it with a str path may be the best workaround.

[daira]

Filed against Twisted as ​https://twistedmatrix.com/trac/ticket/7928.

[daira]

There are actually two bugs here, one in Tahoe-LAFS (we should not be attempting to test Unicode filenames that the filesystem encoding can't represent), and the one in Twisted. The latter only causes a test failure on the branches of Tahoe-LAFS that have Windows inotify-emulation support enabled (because only those branches use Unicode-mode FilePaths).

[daira]

Replying to daira:

There are actually two bugs here, one in Tahoe-LAFS (we should not be attempting to test Unicode filenames that the filesystem encoding can't represent), and the one in Twisted. The latter only causes a test failure on the branches of Tahoe-LAFS that have Windows inotify-emulation support enabled (because only those branches use Unicode-mode FilePaths).

Correction: I had already fixed the first bug on those branches. The remaining problem is that we have this code in encodingutil.py which attempts to determine whether a Unicode-mode FilePath should be used:

use_unicode_filepath = sys.platform == "win32" or hasattr(FilePath, '_asTextPath')

Because of the Twisted bug, for now this needs to be just:

use_unicode_filepath = sys.platform == "win32"

(We won't hit the Twisted bug on Windows because on that platform we use our own INotify emulation which does accept Unicode FilePaths, and which even works on versions of Twisted before the fix to ​https://twistedmatrix.com/trac/ticket/7805.)