#456 closed enhancement (fixed)

it would be nice if the dependency on OpenSSL could be automatically resolved

Reported by: warner Owned by: cgalvan
Priority: major Milestone: 1.6.0
Component: packaging Version: 1.0.0
Keywords: test easy Cc:
Launchpad Bug: 238658


After #438 and foolscap#66 got Foolscap to announce a conditional dependency upon pyopenssl, we found that many of our platforms can't actually build pyopenssl automatically.

  • libssl-dev must be installed, and pyopenssl has no way to declare its dependencies upon non-Python external libraries
  • pyopenssl appears to have syntax errors when compiled against certain versions of libssl. One such error (on an ubuntu/gutsy system) was:
Running pyOpenSSL-0.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-rUEMMk/pyOpenSSL-0.6/egg-dist-tmp-Drvu7p
warning: no previously-included files matching '.cvsignore' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
In file included from src/crypto/crypto.h:19,
                 from src/crypto/crypto.c:13:
src/crypto/netscape_spki.h:24: error: expected specifier-qualifier-list before "NETSCAPE_SPKI"

To work around these for 1.1.0, we're just going to remove tahoe's dependency upon the "secure_connections" feature of Foolscap. This will bypass the check on pyopenssl. We'll figure out a better solution for 1.1.1 .

Change History (36)

comment:1 Changed at 2008-06-09T23:53:36Z by warner

We should also note that many debian platforms don't have valid egg-info files for their pyopenssl installations. dapper and etch have none at all, edgy has an egg-info for py2.5 only, and gutsy has a "pyOpenSSL.egg-info" (as opposed to the "pyOpenSSL-0.6.egg-info" file that we would expect).

So many of our target platforms will be unable to tell that pyopenssl is installed, and are therefore likely to try to build it anyways. This is unfortunate, as a binary debian package of python-openssl or foolscap (with all of its recommended dependencies) should be sufficient.

We might want to revert the foolscap change that declares this dependency.. it may cause more harm than good.

comment:2 Changed at 2008-06-10T16:51:06Z by zooko

The error on the Ubuntu/Gutsy? system turned out to be because I had installed yassl there. Uninstalling yassl made it so that pyOpenSSL could build.

comment:3 Changed at 2008-06-10T16:51:17Z by zooko

The same was true on our dapper buildslave.

comment:4 Changed at 2008-06-18T21:16:19Z by zooko

  • Summary changed from pyopenssl has build problems to pyopenssl depends on a non-Python-packaged thing (OpenSSL)

comment:5 Changed at 2008-06-19T22:44:45Z by warner

So, I don't know how to fix this. setuptools is not APT, and knows nothing about C libraries. I don't think it is likely or reasonable to expect that the pyopenssl authors will put a full copy of !OpenSSL in their source tree.

Personally, I think this is going to be a "root dependency": something that isn't going to get auto-installed, and we should just document the requirement and oblige the developer to get it installed somehow.

comment:6 Changed at 2008-08-11T18:12:39Z by zooko

  • Summary changed from pyopenssl depends on a non-Python-packaged thing (OpenSSL) to it would be nice if the dependency on OpenSSL could be automatically resolved
  • Type changed from defect to enhancement

Changing this from "defect" to "enhancement", and changing title from "pyopenssl depends on non-Python-packaged thing (OpenSSL)" to "it would be nice if the dependency on OpenSSL could be automatically resolved".

We don't know how to solve this nicely, but I would like to leave this ticket open to reflect the fact that I wish it were better.

On Mac OS X, OpenSSL with development headers is installed by default (if you've installed the Mac OS X developer tools), so it isn't a problem on that platform.

Note that we used to have a similar problem with the Crypto++ library, and I "solved" it by including the Crypto++ source code inside the pycryptopp package and building all of it when pycryptopp is built. I doubt that the pyOpenSSL developers will want to do the same thing. (And indeed, this has raised a different problem for pycryptopp: how to facilitate users linking against their own libcryptopp instead of the version of Crypto++ which is included by source code inside pycryptopp. This would be required by Debian to include pycryptopp in Debian, for example.)

Another possible solution, if you aren't on a platform that comes with OpenSSL lib and headers, and if pyOpenSSL isn't willing to bundle OpenSSL source code itself, would be to stop using pyOpenSSL! There are three reasons why this might be a valid future direction: 1. Tahoe actually doesn't need confidentiality or authentication from its transport layer, for the most part. (There are some cases where it does, but perhaps in the future those will go away.) 2. Foolscap might start using a different crypto layer than SSL. 3. Foolscap might start using a Pythonic SSL implementation.

comment:7 Changed at 2008-08-12T15:10:47Z by zooko

  • Owner changed from somebody to zooko
  • Status changed from new to assigned

Oh, here is yet another way that the dependency on OpenSSL could be semi-automatically resolved for some people: if binaries of pyOpenSSL which contained the OpenSSL lib itself were automatically installabe on their platforms. pyOpenSSL bug#238658 says that egenix is now distributing binaries of pyOpenSSL 0.7.

At some point I will investigate whether those pyOpenSSL binaries can satisfy our dependencies on various platforms.

comment:8 Changed at 2008-08-13T15:09:12Z by cgalvan

It is definitely possible to distribute pyOpenSSL eggs along with the OpenSSL binaries, as this is the approach that my company has taken for our python distribution. The problem you may run into is binary incompatibility between different Linux distros, since currently the platform extension for an egg on Linux would be 'linux_x86_64', for example, no matter what specific distro you were on.

comment:9 Changed at 2008-08-13T15:27:55Z by cgalvan

Also, even if you can't get these eggs pushed to PyPi?, you can add dependency links in your setup.py, which would allow you to host them from your own repo.

comment:10 Changed at 2008-08-19T18:12:10Z by zooko

Has your company built eggs for OpenSSL?

comment:11 Changed at 2008-08-19T23:00:45Z by cgalvan

Yes, we have built eggs for OpenSSL, although we mainly use them as build dependencies so that other packages can find them when they are being built.

comment:12 Changed at 2008-08-20T00:44:31Z by zooko

Hm... Would you be interested in supporting binary eggs of pyOpenSSL? By which I mean building them and uploading them to pypi. See this wishlist item on the pyOpenSSL tracker: https://bugs.launchpad.net/pyopenssl/+bug/238658 (please provide binaries)

comment:13 Changed at 2008-08-20T01:47:27Z by cgalvan

Yeah, I just responded to the ticket you mentioned above. At the moment, I'd be able to provide binary(eggs) for win32 a universal egg for OS X 10.3+.

I have also built eggs for some Linux distros(red hat and ubuntu), but they require post-install scripts to be run in order to fix up the rpaths in the binaries, which is not supported natively by setuptools ATM :/ The other problem with providing binary eggs for Linux platforms is that setuptools currently brands all binary linux eggs with <egg-name>-linux-<architecture>.egg, so there would be binary incompatibilities. I have a patch to make the platform branding more specific in the egg names, but it hasn't been looked at yet :/

comment:14 Changed at 2008-08-27T04:13:35Z by cgalvan

I submitted a patch that will aid in providing binaries for win32 and OS X and it is currently being reviewed.

comment:15 Changed at 2008-08-27T16:56:38Z by zooko


comment:16 Changed at 2008-11-05T22:54:44Z by launchpad

  • Launchpad Bug set to 238658

Updating Launchpad bug reference

comment:17 Changed at 2008-12-02T23:19:39Z by cgalvan

While we are waiting for our patches to be accepted to the pyOpenSSL project, I have built an egg for windows of the latest version of pyOpenSSL:


comment:18 Changed at 2008-12-03T02:30:23Z by zooko

  • Owner changed from zooko to cgalvan
  • Status changed from assigned to new

Thanks, Chris! Hm, too bad http://enthought.com is not loading at the moment.

If you get a chance, could you upload that egg to http://testgrid.allmydata.org:8123 and paste the resulting URL into http://allmydata.org/trac/tahoe/wiki/Dependencies ? (According to setup.cfg, this should make it automatically downloaded during builds -- right?)

Also, would you please build eggs for python 2.4 as well?

Thanks again!

comment:19 Changed at 2008-12-03T12:26:09Z by zooko

Okay, I uploaded the pyopenssl egg to allmydata.org, but it turns out the egg doesn't work for me:

$ PYTHONPATH=./pyOpenSSL-0.8-py2.5-win32.egg python -c "import OpenSSL" 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Python25\lib\site-packages\PIL\__init__.py", line 11, in <module>
  File "c:\playground\allmydata\tahoe\installtahoe\trunk-hashedformat\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\SSL.py", line 7, in <module>
  File "c:\playground\allmydata\tahoe\installtahoe\trunk-hashedformat\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\SSL.py", line 6, in __bootstrap__
ImportError: DLL load failed with error code 182

comment:20 Changed at 2008-12-04T12:28:31Z by zooko

Hm, I just tried it again after removing some other pyOpenSSL packages out of my C:\Python25\Lib\site-packages and this time I get a different error message:

$ PYTHONPATH=./pyOpenSSL-0.8-py2.5-win32.egg python -c "import OpenSSL"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Python25\lib\site-packages\PIL\__init__.py", line 11, in <module>
  File "C:\cygwin\tmp\x\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 7, in <module>
  File "C:\cygwin\tmp\x\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 6, in __bootstrap__
ImportError: DLL load failed: The specified module could not be found.

It's strange that the error message changed -- I had hoped that setuptools was arranging to use only the contents of this package.

comment:21 Changed at 2008-12-04T14:05:39Z by zooko

So, if I unzip Chris's egg then I can "import OpenSSL" and it works (actually it fails with permissions errors, and then I "chmod +x *.pyd *.dll", and then it works). But if I rezip that directory (including the chmod +x) back into an egg and load it with PYTHONPATH=./the.egg python -c "import OpenSSL" then I still get this same error: DLL load failed.

See that weirdness with PIL in the stack trace? I removed the "PIL.pth" file I found in my system directory and that weirdness went away. Here's a cleaner stack trace with no PIL, of my attempt to load OpenSSL from the egg which results from me rezipping Chris's egg after chmod +x.

$ PYTHONPATH=./pyOpenSSL-0.8-py2.5-win32.egg python -c "import OpenSSL"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\cygwin\tmp\x\new\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\__init__.py", line 11, in <module>
  File "C:\cygwin\tmp\x\new\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 7, in <module>
  File "C:\cygwin\tmp\x\new\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 6, in __bootstrap__
ImportError: DLL load failed: The specified module could not be found.

comment:22 Changed at 2008-12-04T14:11:43Z by zooko

It fails likewise after being easy_install'ed:

$ easy_install.exe  ./pyOpenSSL-0.8-py2.5-win32.egg 
Processing pyOpenSSL-0.8-py2.5-win32.egg
Copying pyOpenSSL-0.8-py2.5-win32.egg to c:\python25\lib\site-packages
Adding pyOpenSSL 0.8 to easy-install.pth file

Installed c:\python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg
Processing dependencies for pyOpenSSL==0.8
Finished processing dependencies for pyOpenSSL==0.8

Zooko Brillnonywonx@your-fa38fa253f /tmp/x/tmp
$ python -c "import OpenSSL"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg\OpenSSL\__init__.py", line 11, in <module>
  File "c:\Python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 7, in <module>
  File "c:\Python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 6, in __bootstrap__
ImportError: DLL load failed: The specified module could not be found.

comment:23 Changed at 2008-12-04T14:47:01Z by cgalvan

Hm these are some odd errors, I didn't run into any of these when testing on my machine :/ I will see what I can do about this later today when I get back from UNI.

comment:24 Changed at 2008-12-04T16:32:37Z by zooko

Thanks for your help. I hope we can get this running smoothly as soon as possible, and then the imminent Tahoe-1.3.0 release can use it. I've been testing a bunch of other setup/build/install/test/doc improvements on my Windows machine (including your setuptools_trial plugin), but the lack of pyOpenSSL is currently blocking my testing.

comment:25 Changed at 2008-12-04T23:35:21Z by zooko

I can build openssl and pyOpenSSL myself now, using mingw, by applying a patch to openssl-0.9.8i, but the resulting .egg has the same problem, as described over on https://bugs.launchpad.net/pyopenssl/+bug/238658

comment:27 Changed at 2009-01-27T17:32:43Z by zooko

Chris: will you please build a Python 2.6 egg? :-)

comment:28 Changed at 2009-02-05T07:42:26Z by zooko

  • Resolution set to fixed
  • Status changed from new to closed

comment:29 Changed at 2009-03-09T16:56:07Z by zooko

  • Milestone changed from 1.3.1 to 1.3.0

This was fixed for 1.3.0.

comment:30 Changed at 2009-03-18T02:09:17Z by zooko

  • Resolution fixed deleted
  • Status changed from closed to reopened

It looks like at least the Python 2.6 egg is built wrong: #664 (Unable to start.)

comment:31 Changed at 2009-08-06T18:48:00Z by zooko

  • Milestone changed from 1.3.0 to 1.5.1

JP Calderone has built binaries and attached them to the launchpad ticket: https://bugs.launchpad.net/pyopenssl/+bug/238658 . We should test them out and if they work upload them to our web site which serves up Tahoe-LAFS dependencies: http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs .

comment:32 Changed at 2009-08-10T15:51:38Z by zooko

  • Keywords test added

This ticket is just waiting for someone to manually test it (or better yet, program a buildbot to automatically test it). "It" in this case is that that binary .eggs that JP Calderone uploaded to the Launchpad ticket work.

comment:33 Changed at 2009-10-26T20:12:52Z by zooko

  • Keywords easy added

cgalvan: are you planning to do this anytime soon? If you can confirm that the binary eggs that JP Calderone built work, then please close this ticket as "fixed". If you're not planning to do it, in say the next week then please assign this ticket to "nobody" in case someone else wants to pick it up before you get a round to it. Thanks!

comment:34 Changed at 2009-10-27T22:06:04Z by zooko

  • Milestone changed from 1.5.1 to eventually

comment:35 Changed at 2009-10-30T04:00:37Z by cgalvan

  • Resolution set to fixed
  • Status changed from reopened to closed

Sorry for the delay on this one, I tested both eggs out today and I was able to run some basic scripts that use pyOpenSSL so I think it is safe to close this ticket.

comment:36 Changed at 2009-10-30T04:03:49Z by zooko

  • Milestone changed from eventually to 1.6.0

Thanks, Chris!

Note: See TracTickets for help on using tickets.