Magic Folder: Unicode-related test failures

[daira]

​https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Markus%20slackware64%20stable/builds/225/steps/tox/logs/stdio

[ERROR]
Traceback (most recent call last):
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/test_magic_folder.py", line 1273, in setUp
    d = super(MockTest, self).setUp()
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/test_magic_folder.py", line 966, in setUp
    d = self.create_invite_join_magic_folder(u"Alice\u0101", self.local_dir)
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/cli/test_magic_folder.py", line 99, in create_invite_join_magic_folder
    nickname_arg = unicode_to_argv(nickname)
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/util/encodingutil.py", line 123, in unicode_to_argv
    return s.encode(io_encoding)
exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u0101' in position 5: ordinal not in range(128)

allmydata.test.test_magic_folder.MockTest.test_db_basic
allmydata.test.test_magic_folder.MockTest.test_db_persistence
allmydata.test.test_magic_folder.MockTest.test_delete
allmydata.test.test_magic_folder.MockTest.test_delete_and_restore
allmydata.test.test_magic_folder.MockTest.test_errors
allmydata.test.test_magic_folder.MockTest.test_magic_folder
allmydata.test.test_magic_folder.MockTest.test_move_tree
allmydata.test.test_magic_folder.MockTest.test_periodic_full_scan
allmydata.test.test_magic_folder.MockTest.test_persistence
allmydata.test.test_magic_folder.MockTest.test_scan_once_on_startup
allmydata.test.test_magic_folder.MockTest.test_statistics
allmydata.test.test_magic_folder.MockTest.test_write_downloaded_file
===============================================================================
[...]

[ERROR]
Traceback (most recent call last):
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/test_magic_folder.py", line 249, in <lambda>
    d.addCallback(lambda ign: self.do_invite(0, u"Alice\u00F8"))
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/cli/test_magic_folder.py", line 35, in do_invite
    nickname_arg = unicode_to_argv(nickname)
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/util/encodingutil.py", line 123, in unicode_to_argv
    return s.encode(io_encoding)
exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\xf8' in position 5: ordinal not in range(128)

allmydata.test.test_magic_folder.MockTestAliceBob.test_alice_bob
allmydata.test.test_magic_folder.MockTestAliceBob.test_alice_create_bob_update
allmydata.test.test_magic_folder.MockTestAliceBob.test_alice_delete_and_restore
allmydata.test.test_magic_folder.MockTestAliceBob.test_alice_delete_bob_restore
allmydata.test.test_magic_folder.MockTestAliceBob.test_alice_sees_bobs_delete_with_error
allmydata.test.test_magic_folder.MockTestAliceBob.test_download_retry
allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_bob
allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_create_bob_update
allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_delete_and_restore
allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_delete_bob_restore
allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_sees_bobs_delete_with_error
allmydata.test.test_magic_folder.RealTestAliceBob.test_download_retry
===============================================================================

[ERROR]
Traceback (most recent call last):
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/test_magic_folder.py", line 1430, in setUp
    d = super(RealTest, self).setUp()
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/test_magic_folder.py", line 966, in setUp
    d = self.create_invite_join_magic_folder(u"Alice\u0101", self.local_dir)
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/cli/test_magic_folder.py", line 99, in create_invite_join_magic_folder
    nickname_arg = unicode_to_argv(nickname)
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/util/encodingutil.py", line 123, in unicode_to_argv
    return s.encode(io_encoding)
exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u0101' in position 5: ordinal not in range(128)

allmydata.test.test_magic_folder.RealTest.test_db_basic
allmydata.test.test_magic_folder.RealTest.test_db_persistence
allmydata.test.test_magic_folder.RealTest.test_delete
allmydata.test.test_magic_folder.RealTest.test_delete_and_restore
allmydata.test.test_magic_folder.RealTest.test_magic_folder
allmydata.test.test_magic_folder.RealTest.test_move_tree
allmydata.test.test_magic_folder.RealTest.test_persistence
allmydata.test.test_magic_folder.RealTest.test_scan_once_on_startup
===============================================================================

[daira]

The relevant line in cli/test_magic_folder.py is:

    nickname_arg = unicode_to_argv(nickname)

It (and the similar line after it) just needs a fallback if the character isn't representable in the argv encoding.

[warner]

I'll try making that change and see if the buildbot is happy.. if not, I'll reassign it back to daira for further investigation.

[daira]

​https://github.com/tahoe-lafs/tahoe-lafs/pull/356

[Daira Hopwood <daira@…>]

In c9d73a9/trunk:

Fix Unicode-related Magic Folder test failures. fixes #2807

Signed-off-by: Daira Hopwood <daira@…>

[Brian Warner <warner@…>]

In e8c246f/trunk:

Merge daira/2807.magic-folder-unicode-test-fail.1

refs ticket:2807

This allows tests to pass under an odd environment: our OS-X buildslave.
That process is started as a LaunchDaemon? that doesn't provide any
Environment dictionary in the .plist file, so it runs with an empty
environment. On OS-X, this causes filesystem.encoding to default to
"utf-8", but locale.getpreferredencoding() returns "US-ASCII". The tests
previously assumed that any platform which used unicode for the
filesystem pathnames would also use it for sys.argv .

To simulate this without involving launchd, use "unset LANG", and
double-check that locale.getpreferredencoding() has changed.

This fix tolerates such platforms.