[tahoe-dev] PyCrypto build failures (was: 1.9 update: soon! Need help with PyCrypto-2.4!)

[Zooko O'Whielacronx]

Thanks to Vladimir Rusinov and Jimmy Tang for speaking up to explain
that Python 2.4 is still in use. As Josh "Zancas" Wilcox alluded,
Python 2.4 isn't actually the problem! I just assumed that because the
only buildslave which failed was our one Python 2.4 buildslave, and
normally when that one fails it is because someone has written code
that doesn't work on Python 2.4.

However, that's not the case this time. Instead there is a bug in the
newly released PyCrypto (which confusingly happens to be PyCrypto
v2.4) such that it can't be built at all using setuptools,
zc.buildout, or virtualenv:

https://bugs.launchpad.net/pycrypto/+bug/881130

I suspect all of our other buildslaves besides the Python 2.4
buildslave already had PyCrypto installed before the build began so
they didn't attempt to build it.

Zancas reported on IRC that they could work around this bug by running
Tahoe-LAFS's "python setup.py build" a second time after this PyCrypto
build failure occurred.

I'm not sure what to do. From the bug report (PyCrypto #881130), it
sounds like the maintainer of PyCrypto wants the build step to execute
a ./configure script before compiling. That's not something that
setuptools (or zc.buildout or virtualenv, apparently) is designed to
support and I don't know how long it will take for someone to figure
out how to either make setuptools do that or change PyCrypto to not
need that.

The original reporter of the bug, who is the author of "fabric", said
that they're marking fabric as depending on PyCrypto >= 2.1, < 2.4. It
is tempting to do the same. We could change our current dependency on
PyCrypto, which is:

# pycrypto 2.2 doesn't work due to
https://bugs.launchpad.net/pycrypto/+bug/620253
"pycrypto == 2.0.1, == 2.1.0, >= 2.3",

https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/_auto_deps.py?annotate=blame&rev=5156#L45

To say something like:

# pycrypto 2.2 doesn't work due to
https://bugs.launchpad.net/pycrypto/+bug/620253
# pycrypto 2.4 doesn't work due to
https://bugs.launchpad.net/pycrypto/+bug/881130
"pycrypto == 2.0.1, == 2.1.0, >= 2.3, < 2.4",

Or maybe:

"pycrypto == 2.0.1, == 2.1.0, >= 2.3, != 2.4",

If we think the next release of PyCrypto after 2.4 is going to be better.

I'm a bit reluctant to add constraints to our dependencies,
though—they have a way of causing unnecessary hassle. If someone
already has PyCrypto 2.4 installed, for example, then either of these
constraints would force our build system to install a different
version of it, which wouldn't work if that person were on a train at
the time.

Another, complementary approach would be to get someone who has
CentOS/RHEL 5 (calling volunteers!) run "python setup.py bdist_egg",
possibly running it twice in a row (Zancas's work-around), and then
give us the resulting .egg file to be hosted here:

https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html

Then the Tahoe-LAFS build system will download the pre-built egg (if
it can reach our web server) instead of attempting to build PyCrypto
itself.

In any case, I now think that we should continue to support Python 2.4
for now. We have two people who have said it might help them, and we
have one person (FreeStorm) who maintains a Python 2.4 buildslave, and
my main criterion for whether a platform ought to be supported is
whether there is a volunteer who is willing to run a reliable
buildslave for that platform.

Regards,

Zooko