#2807 closed defect (fixed)

Magic Folder: Unicode-related test failures

Reported by: daira Owned by: warner
Priority: normal Milestone: 1.12.0
Component: code-frontend-magic-folder Version: 1.11.0
Keywords: magic-folder unicode Cc:
Launchpad Bug:

Description (last modified by daira)

https://tahoe-lafs.org/buildbot-tahoe-lafs/builders/Markus%20slackware64%20stable/builds/225/steps/tox/logs/stdio

[ERROR]
Traceback (most recent call last):
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/test_magic_folder.py", line 1273, in setUp
    d = super(MockTest, self).setUp()
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/test_magic_folder.py", line 966, in setUp
    d = self.create_invite_join_magic_folder(u"Alice\u0101", self.local_dir)
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/cli/test_magic_folder.py", line 99, in create_invite_join_magic_folder
    nickname_arg = unicode_to_argv(nickname)
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/util/encodingutil.py", line 123, in unicode_to_argv
    return s.encode(io_encoding)
exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u0101' in position 5: ordinal not in range(128)

allmydata.test.test_magic_folder.MockTest.test_db_basic
allmydata.test.test_magic_folder.MockTest.test_db_persistence
allmydata.test.test_magic_folder.MockTest.test_delete
allmydata.test.test_magic_folder.MockTest.test_delete_and_restore
allmydata.test.test_magic_folder.MockTest.test_errors
allmydata.test.test_magic_folder.MockTest.test_magic_folder
allmydata.test.test_magic_folder.MockTest.test_move_tree
allmydata.test.test_magic_folder.MockTest.test_periodic_full_scan
allmydata.test.test_magic_folder.MockTest.test_persistence
allmydata.test.test_magic_folder.MockTest.test_scan_once_on_startup
allmydata.test.test_magic_folder.MockTest.test_statistics
allmydata.test.test_magic_folder.MockTest.test_write_downloaded_file
===============================================================================
[...]

[ERROR]
Traceback (most recent call last):
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/test_magic_folder.py", line 249, in <lambda>
    d.addCallback(lambda ign: self.do_invite(0, u"Alice\u00F8"))
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/cli/test_magic_folder.py", line 35, in do_invite
    nickname_arg = unicode_to_argv(nickname)
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/util/encodingutil.py", line 123, in unicode_to_argv
    return s.encode(io_encoding)
exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\xf8' in position 5: ordinal not in range(128)

allmydata.test.test_magic_folder.MockTestAliceBob.test_alice_bob
allmydata.test.test_magic_folder.MockTestAliceBob.test_alice_create_bob_update
allmydata.test.test_magic_folder.MockTestAliceBob.test_alice_delete_and_restore
allmydata.test.test_magic_folder.MockTestAliceBob.test_alice_delete_bob_restore
allmydata.test.test_magic_folder.MockTestAliceBob.test_alice_sees_bobs_delete_with_error
allmydata.test.test_magic_folder.MockTestAliceBob.test_download_retry
allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_bob
allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_create_bob_update
allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_delete_and_restore
allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_delete_bob_restore
allmydata.test.test_magic_folder.RealTestAliceBob.test_alice_sees_bobs_delete_with_error
allmydata.test.test_magic_folder.RealTestAliceBob.test_download_retry
===============================================================================

[ERROR]
Traceback (most recent call last):
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/test_magic_folder.py", line 1430, in setUp
    d = super(RealTest, self).setUp()
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/test_magic_folder.py", line 966, in setUp
    d = self.create_invite_join_magic_folder(u"Alice\u0101", self.local_dir)
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/test/cli/test_magic_folder.py", line 99, in create_invite_join_magic_folder
    nickname_arg = unicode_to_argv(nickname)
  File "/src/buildslave/tahoe-lafs/Markus_slackware64_stable/build/src/allmydata/util/encodingutil.py", line 123, in unicode_to_argv
    return s.encode(io_encoding)
exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u0101' in position 5: ordinal not in range(128)

allmydata.test.test_magic_folder.RealTest.test_db_basic
allmydata.test.test_magic_folder.RealTest.test_db_persistence
allmydata.test.test_magic_folder.RealTest.test_delete
allmydata.test.test_magic_folder.RealTest.test_delete_and_restore
allmydata.test.test_magic_folder.RealTest.test_magic_folder
allmydata.test.test_magic_folder.RealTest.test_move_tree
allmydata.test.test_magic_folder.RealTest.test_persistence
allmydata.test.test_magic_folder.RealTest.test_scan_once_on_startup
===============================================================================

Change History (8)

comment:1 Changed at 2016-08-16T16:36:35Z by daira

  • Description modified (diff)
  • Status changed from new to assigned

comment:2 Changed at 2016-09-06T17:08:34Z by daira

The relevant line in cli/test_magic_folder.py is:

    nickname_arg = unicode_to_argv(nickname)

It just needs a fallback if the character isn't representable in the argv encoding.

Version 0, edited at 2016-09-06T17:08:34Z by daira (next)

comment:3 Changed at 2016-09-13T17:16:52Z by warner

  • Owner changed from daira to warner
  • Status changed from assigned to new

I'll try making that change and see if the buildbot is happy.. if not, I'll reassign it back to daira for further investigation.

comment:4 Changed at 2016-09-20T16:15:41Z by daira

  • Owner changed from warner to daira
  • Status changed from new to assigned

comment:6 Changed at 2016-09-27T15:13:31Z by daira

  • Owner changed from daira to warner
  • Status changed from assigned to new

comment:7 Changed at 2016-09-27T16:30:16Z by Daira Hopwood <daira@…>

  • Resolution set to fixed
  • Status changed from new to closed

In c9d73a9/trunk:

Fix Unicode-related Magic Folder test failures. fixes #2807

Signed-off-by: Daira Hopwood <daira@…>

comment:8 Changed at 2016-09-27T17:32:37Z by Brian Warner <warner@…>

In e8c246f/trunk:

Merge daira/2807.magic-folder-unicode-test-fail.1

refs ticket:2807

This allows tests to pass under an odd environment: our OS-X buildslave.
That process is started as a LaunchDaemon? that doesn't provide any
Environment dictionary in the .plist file, so it runs with an empty
environment. On OS-X, this causes filesystem.encoding to default to
"utf-8", but locale.getpreferredencoding() returns "US-ASCII". The tests
previously assumed that any platform which used unicode for the
filesystem pathnames would also use it for sys.argv .

To simulate this without involving launchd, use "unset LANG", and
double-check that locale.getpreferredencoding() has changed.

This fix tolerates such platforms.

Note: See TracTickets for help on using tickets.