make tests work under a non-ASCII path

[daira]

Currently when I run tests for a checkout of Tahoe-LAFS (at 9ac392802b6034aefd4c742ea9dd02813f4667d9/trunk) on Linux under a directory with "€" in its name, there are many test failures:

FAILED (skips=5, expectedFailures=3, failures=2, errors=79, successes=1075)

After working around a shallow problem with os.statvfs not accepting a Unicode path, this becomes:

FAILED (skips=5, expectedFailures=3, failures=2, errors=71, successes=1083)

Most of the remaining errors have this traceback:

  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 172, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/home/davidsarah/tahoe/git/€/tahoe-lafs/src/allmydata/scripts/runner.py", line 107, in runner
    config.parseOptions(argv)
  File "/usr/lib/python2.7/dist-packages/twisted/python/usage.py", line 252, in parseOptions
    self._dispatch[optMangled].dispatch(optMangled, arg)
  File "/usr/lib/python2.7/dist-packages/twisted/python/usage.py", line 56, in dispatch
    raise UsageError("Parameter type enforcement failed: %s" % (e,))
twisted.python.usage.UsageError: Parameter type enforcement failed: 'ascii' codec can't encode character u'\u20ac' in position 27: ordinal not in range(128)

[daira]

After fixing a missing call to unicode_to_argv in test_cli.CLITestMixin.do_cli which caused the "Parameter type enforcement failed" errors, this is down to:

FAILED (skips=5, expectedFailures=3, failures=2, errors=9, successes=1145)

[daira]

Another 6 errors in test_auth were easily fixed. That leaves:

[FAIL]
Traceback (most recent call last):
  File "/home/davidsarah/tahoe/git/€/tahoe-lafs/src/allmydata/test/test_system.py", line 1384, in _test_runner
    self.failUnlessEqual(len(descriptions), 30)
  File "/usr/lib/python2.7/dist-packages/twisted/trial/_synctest.py", line 356, in assertEqual
    % (msg, pformat(first), pformat(second)))
twisted.trial.unittest.FailTest: not equal:
a = 0
b = 30


allmydata.test.test_system.SystemTest.test_filesystem
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 172, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/home/davidsarah/tahoe/git/€/tahoe-lafs/src/allmydata/scripts/runner.py", line 141, in runner
    rc = cli.dispatch[command](so)
  File "/home/davidsarah/tahoe/git/€/tahoe-lafs/src/allmydata/scripts/cli.py", line 531, in get
    (options.from_file, options.to_file)
exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 74: ordinal not in range(128)

allmydata.test.test_cli.Mv.test_mv_behavior
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 172, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/home/davidsarah/tahoe/git/€/tahoe-lafs/src/allmydata/scripts/runner.py", line 107, in runner
    config.parseOptions(argv)
  File "/usr/lib/python2.7/dist-packages/twisted/python/usage.py", line 252, in parseOptions
    self._dispatch[optMangled].dispatch(optMangled, arg)
  File "/usr/lib/python2.7/dist-packages/twisted/python/usage.py", line 56, in dispatch
    raise UsageError("Parameter type enforcement failed: %s" % (e,))
twisted.python.usage.UsageError: Parameter type enforcement failed: 'ascii' codec can't encode character u'\u20ac' in position 27: ordinal not in range(128)

allmydata.test.test_deepcheck.DeepCheckWebGood.test_good
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/home/davidsarah/tahoe/git/€/tahoe-lafs/src/allmydata/test/test_mutable.py", line 3150, in _debug
    oneshare = shares[0] # all shares should be MDMF
exceptions.IndexError: list index out of range

allmydata.test.test_mutable.Version.test_debug
-------------------------------------------------------------------------------
Ran 1164 tests in 288.011s

FAILED (skips=5, expectedFailures=3, failures=2, errors=3, successes=1151)

[daira]

The current work-in-progress branch for this ticket is ​https://github.com/tahoe-lafs/tahoe-lafs/commits/2375.tests-non-ascii.2.

[daira]

The current branch is now ​https://github.com/tahoe-lafs/tahoe-lafs/commits/2375.tests-non-ascii.3. Still not ready for review.

[warner]

The first three patches from that branch look good to me, although it might be nice to squash the first two together. The WIP patch looks like it's headed in the right direction: the non-tests parts that I looked at seemed reasonable.

[daira]

Not ready for 1.10.1.

[warner]

Milestone renamed

[warner]

moving most tickets from 1.12 to 1.13 so we can release 1.12 with magic-folders

[exarkun]

Moving open issues out of closed milestones.

[meejah]

Ticket retargeted after milestone closed