os.path.abspath() sometimes returns str instead of unicode?

[zooko]

Python 2.7 just came out, and reading through the changes led me to:

​http://bugs.python.org/issue3426

We use os.path.abspath() in a few places where its argument is unicode.

I also noticed:

​http://bugs.python.org/issue5827

but we don't currently use os.path.normpath().

How come our unit tests don't show this problem happening to us?

[davidsarah]

• util.fileutil, test.test_util: add abspath_expanduser_unicode function, to work around <​http://bugs.python.org/issue3426>. util.encodingutil: add a convenience function argv_to_abspath. * Replace uses of os.path.abspath with abspath_expanduser_unicode where necessary. This makes basedir paths consistently represented as Unicode.

[davidsarah]

Our tests might not show this for any of the following reasons:

• We don't test or support Unicode paths for basedirs, which account for most of our uses of os.path.abspath.
• Our Unicode tests for the CLI are a bit patchy; they only test Unicode paths for some commands.
• When running the test suite, the current directory is the _trial_temp directory, which is at an ASCII path even if the test creates files and subdirectories at non-ASCII paths.
• The bug in os.path.abspath only occurs when none of the resulting path components come from the input string the input path is relative and the current directory path is non-ASCII.
• Even when os.path.abspath returns a bytestring, it will often get coerced to a Unicode string (not necessarily the correct one!) by the default Python coercions.

Note that while attachment:abspath-and-unicode-basedirs.dpatch​ makes most of the changes needed to support Unicode paths for basedirs, you still can't, at least on Windows, build or test a Tahoe source tree at an arbitrary Unicode path. (That would require changes to Python, darcs, setuptools, Twisted trial, etc., and is probably not a realistic aim.)

[davidsarah]

The changesets relevant to this ticket on the ticket798 branch are: [4598/ticket798] [4599/ticket798] [4601/ticket798] [4604/ticket798] [4606/ticket798] [4607/ticket798]

[davidsarah]

The changesets relevant to this ticket on trunk are 11b18824c7ff3237 618db4867c68a6f9 f036dfaa4bfb9f90 2b33704d613da8d1 013954c2659db71f d3a8ef762f74b925 02199db615047ea1.

[zooko]

Okay I looked over these patches a bit and didn't see anything obviously wrong. It was a pretty quick and lazy review, but better than nothing.