compilation errors with pyOpenSSL >= 0.13 on RHEL5 systems (and clones)

[davidsarah]

Reported by Jimmy Tang:

[jtang@gateway allmydata-tahoe-1.9.0-SUMO]$ python setup.py build
running darcsver
setup.py darcsver: using extant version file ['src/allmydata/_version.py']
running develop
Not found: ../tahoe-deps
running egg_info
writing requirements to src/allmydata_tahoe.egg-info/requires.txt
writing src/allmydata_tahoe.egg-info/PKG-INFO
writing top-level names to src/allmydata_tahoe.egg-info/top_level.txt
writing dependency_links to src/allmydata_tahoe.egg-info/dependency_links.txt
writing entry points to src/allmydata_tahoe.egg-info/entry_points.txt
reading manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt'
writing manifest file 'src/allmydata_tahoe.egg-info/SOURCES.txt'
running build_ext
Processing setuptools_darcs-1.2.12.egg
removing '/home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages/setuptools_darcs-1.2.12.egg'
(and everything under it)
Copying setuptools_darcs-1.2.12.egg to
/home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages
setuptools-darcs 1.2.12 is already the active version in easy-install.pth

Installed /home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages/setuptools_darcs-1.2.12.egg
Creating /home/jtang/allmydata-tahoe-1.9.0-SUMO/support/lib/python2.4/site-packages/allmydata-tahoe.egg-link
(link to src)
allmydata-tahoe 1.9.0 is already the active version in easy-install.pth
Installing tahoe script to support/bin

Installed /home/jtang/allmydata-tahoe-1.9.0-SUMO/src
Processing dependencies for allmydata-tahoe==1.9.0
Searching for pyOpenSSL
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-dep-sdists/
Reading http://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/
Best match: pyOpenSSL 0.13
Processing pyOpenSSL-0.13.tar.gz
Running pyOpenSSL-0.13/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-gr--LP/pyOpenSSL-0.13/egg-dist-tmp-YkOeMn
warning: no previously-included files matching '*.pyc' found anywhere
in distribution
OpenSSL/ssl/connection.c: In function 'ssl_Connection_set_context':
OpenSSL/ssl/connection.c:289: warning: implicit declaration of
function 'SSL_set_SSL_CTX'
OpenSSL/ssl/connection.c: In function 'ssl_Connection_get_servername':
OpenSSL/ssl/connection.c:313: error: 'TLSEXT_NAMETYPE_host_name'
undeclared (first use in this function)
OpenSSL/ssl/connection.c:313: error: (Each undeclared identifier is
reported only once
OpenSSL/ssl/connection.c:313: error: for each function it appears in.)
OpenSSL/ssl/connection.c:320: warning: implicit declaration of
function 'SSL_get_servername'
OpenSSL/ssl/connection.c:320: warning: assignment makes pointer from
integer without a cast
OpenSSL/ssl/connection.c: In function 'ssl_Connection_set_tlsext_host_name':
OpenSSL/ssl/connection.c:346: warning: implicit declaration of
function 'SSL_set_tlsext_host_name'
error: Setup script exited with error: command 'gcc' failed with exit status 1

as far as I know it's due to the pyopenssl needing newer features of openssl -- see

​https://bugs.launchpad.net/pyopenssl/+bug/845445

Doing this somewhat makes the problem go away on RHEL5

diff --git a/src/allmydata/_auto_deps.py b/src/allmydata/_auto_deps.py
index bee1b5d..57d82b1 100644
--- a/src/allmydata/_auto_deps.py
+++ b/src/allmydata/_auto_deps.py
@@ -38,14 +38,13 @@ install_requires = [
     # or foolscap switching from pyOpenSSL to a different crypto library, we
     # need to update this declaration here.
     "foolscap >= 0.6.1",
-    "pyOpenSSL",
+    "pyOpenSSL <= 0.11",

the tests all pass after making the change.

[davidsarah]

On 15/12/11 13:51, Jimmy Tang wrote:

Is there a compelling reason for using newer versions of pyopenssl ?

No, but we dislike having to exclude newer versions of dependencies; it means that security updates don't get picked up, for example.

[zooko]

One solution to this for Tahoe-LAFS would be to build an egg to put in the pyOpenSSL column of the Python 2.4 rows of https://tahoe-lafs.org/source/tahoe-lafs/deps/tahoe-lafs-dep-eggs/README.html. If someone with Python 2.4 linux could provide us with such an egg, that would probably fix this problem. (By having the build system detect and download that egg instead of attempting to compile pyOpenSSL from source.)

[zooko]

This is not a critical problem, but if there is an easy way to fix it for Tahoe-LAFS v1.9.1, I would be willing to do a little work on that. The next step is we need a volunteer who loves and/or operates RHEL to provide us with a precompiled egg and to promise us that he didn't insert any backdoors or allow anyone else to do so.

[zooko]

Jimmy: would you be willing to build a binary package with Python 2.4 on your platform and give it to us?

[jcftang]

Zooko: If you are happy with me submitting a pyopenssl 0.13 egg which links to a newer version of openssl than the stock version in rhel5 and the stock python 2.4 then I can cook one up later on in the day and upload it somewhere, just point me to some instructions and documentation on how you would like it done.

just note that my version of openssl is installed in a none-standard location, so the egg might not behave too well if any new functionality is used that requires the newer features of the updated openssl installation.

The above solution doesn't sound like a runner to me, I think trying to convince the pyopenssl people that rhel5 is still a valid and supported platform is a better way of getting fixed, but this isn't clear to me if rhel5 is supported or not on for pyopenssl 0.13

edit: rhel 5.8 recently got released, rhel5 will be around till at least 2014, I would expect people to migrate away from it en-masse in 2013.

[zooko]

jcftang: it appears to me that when you build a pyOpenSSL egg then it includes a copy of libcrypto.so in the resulting .egg, so that egg will work on a platform even if that version of libcrypto.so is not the standard one for that platform. Please try it! Let's see. :-)

I'm not sure what you mean about the pyOpenSSL people "supporting" RHEL 5. Jean-Paul "exarkun" Calderone is the the most active contributor to pyOpenSSL, and he said something on ​pyOpenSSL ticket 845445 that made it sound like he was willing to work on this, but lacked "a good environment set up to be able to easily test against specific versions of OpenSSL". Maybe we could offer to help him set up such an environment?

Also helping them build automation that produces .eggs automatically on different platforms would certainly help: ​pyOpenSSL ticket 668996.

[jcftang]

zooko: I did not realise that building python eggs included a compiled binary blob, anyway as requested, attached to this ticket is an egg for rhel5.7 (SL5.7 to be exact), it was built using openssl 0.98o. Please test!

Also apologies for not reading ticket pyopenssl Bug #845445 more thoroughly, it looks like they see the problem which is good :).

Under linux I've been using a home brew build system (which allows multiple versions of software to be installed, for a cluster compute environment) but it isn't very mature, it may be of interest. This is how I install alternate versions of openssl, python, whatever for testing and deployment. This may be better discussed on the mailing list if there is interest.

I might be able to help setup something to automatically build some eggs for submission, but I would need to check for permission with the guys in work first before I can commit to anything.

[davidsarah]

#1671 was a duplicate. The problem also happens on CentOS (which is a Redhat derivative).

[davidsarah]

This is pyOpenSSL's bug rather than our bug, therefore, it does not need to be in milestone 1.9.2. I'm leaving this open rather than marking it invalid just because we will need to make sure that an egg of the fixed pyOpenSSL is available.