#1065 closed defect (fixed)

provide binary eggs of PyCrypto for our supported platforms

Reported by: zooko Owned by: somebody
Priority: major Milestone: soon (release n/a)
Component: packaging Version: 1.6.1
Keywords: binaries windows setuptools pycrypto-lib Cc:
Launchpad Bug:

Description

Now that we require PyCrypto (see #953), we need to provide binary eggs of it so that people can install Tahoe-LAFS on platforms that don't have a working C compiler. Here is a buildbot that is currently demonstrating how this fails:

http://tahoe-lafs.org/buildbot/builders/Soultcer%20Win7-64%20py2.6

For example, a failing build is:

http://tahoe-lafs.org/buildbot/builders/Soultcer%20Win7-64%20py2.6/builds/47/steps/build/logs/stdio

which ends with

Searching for pycrypto>=2.0.1
Reading http://pypi.python.org/simple/pycrypto/
Reading http://pycrypto.sourceforge.net
Reading http://www.amk.ca/python/code/crypto
Best match: pycrypto 2.1.0
Downloading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/pycrypto-2.1.0.tar.bz2
Processing pycrypto-2.1.0.tar.bz2
Running pycrypto-2.1.0\setup.py -q bdist_egg --dist-dir c:\users\buildbot\appdata\local\temp\easy_install-ynvpn2\pycrypto-2.1.0\egg-dist-tmp-k5u7tq
Failure to get the list of managed files from darcs -- if you are building a package with 'setup.py sdist', 'setup.py bdist_egg', or other package-building commands, then the resulting package might be missing some files.  If you are not building a package then you can ignore this warning.
warning: GMP library not found; Not building Crypto.PublicKey._fastmath.
error: Setup script exited with error: Unable to find vcvarsall.bat

To close this ticket, make it so that that buildbot succeeds at installing Tahoe-LAFS even if its owner (soultcer) does not install any new tools such as a C compiler nor packages such as PyCrypto.

The way to do that is to build or acquire a binary egg of PyCrypto for that platform (Windows 7 64 bit) and host it here: http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/ (or you can host it on the PyPI site for PyCrypto, which will also be examined to find candidate eggs).

Change History (12)

comment:1 Changed at 2010-06-04T03:37:03Z by zooko

  • Component changed from unknown to packaging
  • Keywords windows setuptools pycrypto-lib added
  • Owner changed from nobody to somebody
  • Summary changed from host binary eggs of PyCrypto for our supported platforms to provide binary eggs of PyCrypto for our supported platforms

comment:2 Changed at 2010-06-04T03:43:42Z by zooko

Here is the launchpad ticket where I request that the maintainers of PyCrypto? provide binary eggs: https://bugs.launchpad.net/tahoe-lafs/+bug/589511

comment:3 Changed at 2010-08-10T04:50:46Z by zooko

The maintainer of PyCrypto, Dwayne Litzenberger, replied with "Won't Fix":

I have no intention of distributing PyCrypto? binaries any time soon. Sorry.

comment:4 Changed at 2010-08-27T20:18:46Z by davidsarah

This problem has been exacerbated by this bug in PyCrypto 2.2 (fixed in 2.3). If we were to bump the PyCrypto version requirement in _auto_deps.py to 2.3, systems with no compiler would fail to upgrade, even if they already had a working PyCrypto egg (say 2.0.1 or 2.1, which are downloadable from http://www.voidspace.org.uk/python/modules.shtml#pycrypto ).

Even if we don't bump the version requirement, a user whose system has no working compiler may run into http://bugs.python.org/setuptools/issue83 (this happened to bj0). #717 and #1015 are also contributing factors here.

comment:5 Changed at 2010-08-27T20:39:16Z by davidsarah

According to setuptools docs, we can write the PyCrypto dependency as "pycrypto == 2.0.1, == 2.1, >= 2.3".

comment:6 Changed at 2010-08-29T22:40:29Z by zooko

Well, I've never used disjunctive version requirements like that before, but let's try it. I'll commit it to ticket798 and have the buildbots build it.

comment:7 follow-up: Changed at 2010-08-30T03:52:51Z by davidsarah

This was actually committed to trunk, in 57b96c5e2f851cca. It doesn't seem to have caused any new buildbot failures.

comment:8 in reply to: ↑ 7 Changed at 2010-08-30T20:11:52Z by davidsarah

Replying to davidsarah:

This was actually committed to trunk, in 57b96c5e2f851cca. It doesn't seem to have caused any new buildbot failures.

Sigh, spoke too soon (log for the test step on 'Ruben Fedora'):

Searching for pycrypto==2.1.0
Best match: pycrypto 2.1.0
Processing pycrypto-2.1.0-py2.7-linux-x86_64.egg
pycrypto 2.1.0 is already the active version in easy-install.pth

Using /home/buildbot/tahoe/Ruben Fedora/build/support/lib/python2.7/
site-packages/pycrypto-2.1.0-py2.7-linux-x86_64.egg
[...]
running trial
[...]
pkg_resources.VersionConflict: (pycrypto 2.2 (/usr/lib64/python2.7/
site-packages), Requirement.parse('pycrypto==2.0.1,==2.1,>=2.3'))

I believe this is due to the same underlying issue as #1137, that is, the handling of sys.path by the setuptools-installed site.py and .pth files being broken. If the path handling were working correctly, it should not have been possible that "pycrypto 2.1.0 is already the active version in easy-install.pth", but we end up importing pycrypto 2.2 when running trial.

Note that we have the same problem with foolscap on 'Shawn jaunty':

Searching for foolscap==0.5.1
Best match: foolscap 0.5.1
Processing foolscap-0.5.1-py2.6.egg
foolscap 0.5.1 is already the active version in easy-install.pth
Installing flappclient script to support/bin
Installing flogtool script to support/bin
Installing flappserver script to support/bin

Using /var/lib/buildbot/tahoe/mordsith/build/support/lib/python2.6/
site-packages/foolscap-0.5.1-py2.6.egg
[...]
running trial
[...]
pkg_resources.VersionConflict: (foolscap 0.4.2 (/usr/local/lib/python2.6/
dist-packages/foolscap-0.4.2-py2.6.egg),
Requirement.parse('foolscap[secure_connections]>=0.5.1'))

So this is not specific to pycrypto or due to the use of a disjunctive requirement. I think that we should keep the pycrypto requirement as "pycrypto == 2.0.1, == 2.1, >= 2.3" for 1.8.0c3, since it at least fails with a message that could give the user a hint that they need to delete the pycrypto 2.2 installation.

comment:9 Changed at 2010-08-30T20:22:10Z by davidsarah

The using-wrong-version-of-dependency problem is #1190.

comment:10 Changed at 2010-10-31T06:08:52Z by zooko

  • Keywords binaries added
  • Milestone changed from soon (release n/a) to 1.8.1

Since the newest version of PyCrypto listed on http://pypi.python.org/pypi/pycrypto at the moment is 2.3, then we need to build bdist_egg's of PyCrypto 2.3 for all platforms which we intend for people to be able to install Tahoe-LAFS v1.8.1 without having a C compiler and Python headers installed. (Note that the upstream maintainer of PyCrypto will not provide binaries: https://bugs.launchpad.net/pycrypto/+bug/589511 .)

Sadly, if we do all the work to build and host such binary eggs, and then a new version of PyCrypto gets uploaded to !PyPI in source form, then our build scripts will immediately start refusing to use the PyCrypto 2.3 binaries that we built. Fixing this so that our build scripts would continue to use the PyCrypto 2.3 binaries even if a newer version of PyCrypto is known to them is the subject of #1233.

comment:11 Changed at 2011-01-06T21:22:01Z by warner

  • Milestone changed from 1.8.1 to soon (release n/a)

comment:12 Changed at 2011-01-17T07:43:00Z by zooko

  • Resolution set to fixed
  • Status changed from new to closed

We now host binary eggs of PyCrypto for many fine platforms. See http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html for details.

Note: See TracTickets for help on using tickets.