id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	launchpad_bug
2739	"windows vs len(""pycryptopp-0.7.1. 869544967005693312591928092448767568728501330214"")"	warner	daira	"(moved from #1582, specifically comment:28:ticket:1582 and comment:38:ticket:1582)

Windows machines have problems building pycryptopp, because the length of the version string (65 chars when turned into a directory, including the package name, connecting hyphen, and version) causes pathnames to exceed the maximum allowable on windows. Zooko points out that this should be considered a bug in setuptools, since building *any* package would fail if the parent directory name is sufficiently long.

I spent the day getting a windows VM set up, and found that the situation is indeed dire when building from a directory with a long name. I used my `spake2` package as a test case, because it is pure-python and has a very short package+version name.

The base directory is `C:\Users\$USERNAME\$AAAA`, where ""$USERNAME"" is 18 characters long.

I started with ""$AAAA"" at 200 characters long, so the total pathname up to this point is 2+1+18+1+200= 221. Inside that is the ""spake2-0.3"" directory, bringing pwd to 232 chars. From a directory of that length, python cannot even import the `versioneer.py` that's sitting right next to setup.py (it can import `setuptools` and other packages that are in the much-shorter `C:\Python27\Lib` directory, but nothing that involves long `sys.path` entries). This is unrelated to setuptools, and if it is to be considered a bug, it would be assigned to python itself.

It is not possible to unpack the pycryptopp tarball into this directory. Moving one that was unpacked elsewhere works, but then even basic windows commands like `dir` fail, with an error like ""The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters"". (note that I'm using `PowerShell` on a Windows 2012 Server system).


When we move to $AAAA=150 chars (for a total length of 182 chars), versioneer can be imported, but a `setup.py bdist_wheel` fails as it attempts to copy the egg-info data into place, with an error like the one seen in comment:27:ticket:1582 . The chain of responsibility is:

* setuptools.command.install_egg_info.copytree()
* setuptools.archive_util.unpack_archive()
* setuptools.archive_util.unpack_directory()
* shutil.copyfile()

`shutil.copyfile()` is handed a `dst=` argument of `build\bdist.win32\wheel\.\spake2-0.3-py2.7.egg-info\dependency_links.txt`, which is a relative pathname of len=72. It then attempts a regular `open(dst, 'wb')`, which fails with errno 2 (No such file or directory).

I'm not sure what setuptools could do differently here. Is there something other than `open()` that python code is supposed to use on windows? I tried making `open` use `os.path.abspath()`, thinking that a prefix of `C:\` might trigger a differently-capable syscall, but that didn't seem to help.

In an AAAA=145 directory (total length 167 chars), spake2-0.3 builds a wheel with no problem.

Note that the package version string appears twice: once in the `copyfile()` argument, and a second time in the implicit CWD (where the source tarball was unpacked). So every extra character in the version string means two characters in the absolute filename.

As a workaround, I found that it *is* possible to build a pycryptopp wheel (using `python setup.py bdist_wheel`) if the parent directory is 2+1+18+1+25= 47 characters or less (the cutoff is somewhere between 47 and 52 characters).

Appveyor.com (the CI service I've been trying to get working) unpacks the source tree into `C:\projects\tahoe-lafs`. If we used them to build pycryptopp wheels from a git checkout, they'd get `C:\projects\pycryptopp`, which would be short enough to work. Technically, we really only need one machine to run `setup.py bdist_wheel`, then everybody else can install the pre-compiled wheels.

Until we have pre-compiled wheels available, windows devs will be building wheels as a side-effect of `pip install`, which will use some temp directory for the purpose. I don't yet know how the temp directory is created (how long it is, and whether it lives under the user's home directory, or somewhere unaffected by the length of their username like /tmp).

So maybe publishing pre-compiled pycryptopp wheels (built on appveyor from git, not from a tarball) will be enough to allow windows devs of all username-lengths to build tahoe.
"	defect	closed	normal	undecided	packaging	1.10.2	wontfix	pycryptopp packaging windows