[tahoe-lafs-trac-stream] [Tahoe-LAFS] #2739: windows vs len("pycryptopp-0.7.1.869544967005693312591928092448767568728501330214")

[Tahoe-LAFS]

#2739: windows vs
len("pycryptopp-0.7.1.869544967005693312591928092448767568728501330214")
-----------------------+---------------------------
 Reporter:  warner     |          Owner:  daira
     Type:  defect     |         Status:  new
 Priority:  normal     |      Milestone:  undecided
Component:  packaging  |        Version:  1.10.2
 Keywords:             |  Launchpad Bug:
-----------------------+---------------------------
 (moved from #1582, specifically comment:28:ticket:1582 and
 comment:38:ticket:1582)

 Windows machines have problems building pycryptopp, because the length of
 the version string (65 chars when turned into a directory, including the
 package name, connecting hyphen, and version) causes pathnames to exceed
 the maximum allowable on windows. Zooko points out that this should be
 considered a bug in setuptools, since building *any* package would fail if
 the parent directory name is sufficiently long.

 I spent the day getting a windows VM set up, and found that the situation
 is indeed dire when building from a directory with a long name. I used my
 `spake2` package as a test case, because it is pure-python and has a very
 short package+version name.

 The base directory is `C:\Users\$USERNAME\$AAAA`, where "$USERNAME" is 18
 characters long.

 I started with "$AAAA" at 200 characters long, so the total pathname up to
 this point is 2+1+18+1+200= 221. Inside that is the "spake2-0.3"
 directory, bringing pwd to 232 chars. From a directory of that length,
 python cannot even import the `versioneer.py` that's sitting right next to
 setup.py (it can import `setuptools` and other packages that are in the
 much-shorter `C:\Python27\Lib` directory, but nothing that involves long
 `sys.path` entries). This is unrelated to setuptools, and if it is to be
 considered a bug, it would be assigned to python itself.

 It is not possible to unpack the pycryptopp tarball into this directory.
 Moving one that was unpacked elsewhere works, but then even basic windows
 commands like `dir` fail, with an error like "The specified path, file
 name, or both are too long. The fully qualified file name must be less
 than 260 characters, and the directory name must be less than 248
 characters". (note that I'm using `PowerShell` on a Windows 2012 Server
 system).


 When we move to $AAAA=150 chars (for a total length of 182 chars),
 versioneer can be imported, but a `setup.py bdist_wheel` fails as it
 attempts to copy the egg-info data into place, with an error like the one
 seen in comment:27:ticket:1582 . The chain of responsibility is:

 * setuptools.command.install_egg_info.copytree()
 * setuptools.archive_util.unpack_archive()
 * setuptools.archive_util.unpack_directory()
 * shutil.copyfile()

 `shutil.copyfile()` is handed a `dst=` argument of
 `build\bdist.win32\wheel\.\spake2-0.3-py2.7.egg-
 info\dependency_links.txt`, which is a relative pathname of len=72. It
 then attempts a regular `open(dst, 'wb')`, which fails with errno 2 (No
 such file or directory).

 I'm not sure what setuptools could do differently here. Is there something
 other than `open()` that python code is supposed to use on windows? I
 tried making `open` use `os.path.abspath()`, thinking that a prefix of
 `C:\` might trigger a differently-capable syscall, but that didn't seem to
 help.

 In an AAAA=145 directory (total length 167 chars), spake2-0.3 builds a
 wheel with no problem.

 Note that the package version string appears twice: once in the
 `copyfile()` argument, and a second time in the implicit CWD (where the
 source tarball was unpacked). So every extra character in the version
 string means two characters in the absolute filename.

 As a workaround, I found that it *is* possible to build a pycryptopp wheel
 (using `python setup.py bdist_wheel`) if the parent directory is
 2+1+18+1+25= 47 characters or less (the cutoff is somewhere between 47 and
 52 characters).

 Appveyor.com (the CI service I've been trying to get working) unpacks the
 source tree into `C:\projects\tahoe-lafs`. If we used them to build
 pycryptopp wheels from a git checkout, they'd get
 `C:\projects\pycryptopp`, which would be short enough to work.
 Technically, we really only need one machine to run `setup.py
 bdist_wheel`, then everybody else can install the pre-compiled wheels.

 Until we have pre-compiled wheels available, windows devs will be building
 wheels as a side-effect of `pip install`, which will use some temp
 directory for the purpose. I don't yet know how the temp directory is
 created (how long it is, and whether it lives under the user's home
 directory, or somewhere unaffected by the length of their username like
 /tmp).

 So maybe publishing pre-compiled pycryptopp wheels (built on appveyor from
 git, not from a tarball) will be enough to allow windows devs of all
 username-lengths to build tahoe.

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2739>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage