it would be nice if the dependency on OpenSSL could be automatically resolved

[warner]

After #438 and ​foolscap#66 got Foolscap to announce a conditional dependency upon pyopenssl, we found that many of our platforms can't actually build pyopenssl automatically.

• libssl-dev must be installed, and pyopenssl has no way to declare its dependencies upon non-Python external libraries
• pyopenssl appears to have syntax errors when compiled against certain versions of libssl. One such error (on an ubuntu/gutsy system) was:

Running pyOpenSSL-0.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-rUEMMk/pyOpenSSL-0.6/egg-dist-tmp-Drvu7p
warning: no previously-included files matching '.cvsignore' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
In file included from src/crypto/crypto.h:19,
                 from src/crypto/crypto.c:13:
src/crypto/netscape_spki.h:24: error: expected specifier-qualifier-list before "NETSCAPE_SPKI"

To work around these for 1.1.0, we're just going to remove tahoe's dependency upon the "secure_connections" feature of Foolscap. This will bypass the check on pyopenssl. We'll figure out a better solution for 1.1.1 .

[warner]

We should also note that many debian platforms don't have valid egg-info files for their pyopenssl installations. dapper and etch have none at all, edgy has an egg-info for py2.5 only, and gutsy has a "pyOpenSSL.egg-info" (as opposed to the "pyOpenSSL-0.6.egg-info" file that we would expect).

So many of our target platforms will be unable to tell that pyopenssl is installed, and are therefore likely to try to build it anyways. This is unfortunate, as a binary debian package of python-openssl or foolscap (with all of its recommended dependencies) should be sufficient.

We might want to revert the foolscap change that declares this dependency.. it may cause more harm than good.

[zooko]

The error on the Ubuntu/Gutsy? system turned out to be because I had installed ​yassl there. Uninstalling yassl made it so that pyOpenSSL could build.

[zooko]

The same was true on our dapper buildslave.

[warner]

So, I don't know how to fix this. setuptools is not APT, and knows nothing about C libraries. I don't think it is likely or reasonable to expect that the pyopenssl authors will put a full copy of !OpenSSL in their source tree.

Personally, I think this is going to be a "root dependency": something that isn't going to get auto-installed, and we should just document the requirement and oblige the developer to get it installed somehow.

[zooko]

Changing this from "defect" to "enhancement", and changing title from "pyopenssl depends on non-Python-packaged thing (OpenSSL)" to "it would be nice if the dependency on OpenSSL could be automatically resolved".

We don't know how to solve this nicely, but I would like to leave this ticket open to reflect the fact that I wish it were better.

On Mac OS X, OpenSSL with development headers is installed by default (if you've installed the Mac OS X developer tools), so it isn't a problem on that platform.

Note that we used to have a similar problem with the Crypto++ library, and I "solved" it by including the Crypto++ source code inside the pycryptopp package and building all of it when pycryptopp is built. I doubt that the pyOpenSSL developers will want to do the same thing. (And indeed, this has raised a different problem for pycryptopp: how to facilitate users linking against their own libcryptopp instead of the version of Crypto++ which is included by source code inside pycryptopp. This would be required by Debian to include pycryptopp in Debian, for example.)

Another possible solution, if you aren't on a platform that comes with OpenSSL lib and headers, and if pyOpenSSL isn't willing to bundle OpenSSL source code itself, would be to stop using pyOpenSSL! There are three reasons why this might be a valid future direction: 1. Tahoe actually doesn't need confidentiality or authentication from its transport layer, for the most part. (There are some cases where it does, but perhaps in the future those will go away.) 2. Foolscap might start using a different crypto layer than SSL. 3. Foolscap might start using a Pythonic SSL implementation.

[zooko]

Oh, here is yet another way that the dependency on OpenSSL could be semi-automatically resolved for some people: if binaries of pyOpenSSL which contained the OpenSSL lib itself were automatically installabe on their platforms. ​pyOpenSSL bug#238658 says that egenix is now distributing binaries of pyOpenSSL 0.7.

At some point I will investigate whether those pyOpenSSL binaries can satisfy our dependencies on various platforms.

[cgalvan]

It is definitely possible to distribute pyOpenSSL eggs along with the OpenSSL binaries, as this is the approach that my company has taken for our python distribution. The problem you may run into is binary incompatibility between different Linux distros, since currently the platform extension for an egg on Linux would be 'linux_x86_64', for example, no matter what specific distro you were on.

[cgalvan]

Also, even if you can't get these eggs pushed to PyPi?, you can add dependency links in your setup.py, which would allow you to host them from your own repo.

[zooko]

Has your company built eggs for OpenSSL?

[cgalvan]

Yes, we have built eggs for OpenSSL, although we mainly use them as build dependencies so that other packages can find them when they are being built.

[zooko]

Hm... Would you be interested in supporting binary eggs of pyOpenSSL? By which I mean building them and uploading them to pypi. See this wishlist item on the pyOpenSSL tracker: ​https://bugs.launchpad.net/pyopenssl/+bug/238658 (please provide binaries)

[cgalvan]

Yeah, I just responded to the ticket you mentioned above. At the moment, I'd be able to provide binary(eggs) for win32 a universal egg for OS X 10.3+.

I have also built eggs for some Linux distros(red hat and ubuntu), but they require post-install scripts to be run in order to fix up the rpaths in the binaries, which is not supported natively by setuptools ATM :/ The other problem with providing binary eggs for Linux platforms is that setuptools currently brands all binary linux eggs with <egg-name>-linux-<architecture>.egg, so there would be binary incompatibilities. I have a patch to make the platform branding more specific in the egg names, but it hasn't been looked at yet :/

[cgalvan]

I submitted a patch that will aid in providing binaries for win32 and OS X and it is currently being reviewed.

[zooko]

Great!

[launchpad]

Updating Launchpad bug reference

[cgalvan]

While we are waiting for our patches to be accepted to the pyOpenSSL project, I have built an egg for windows of the latest version of pyOpenSSL:

​http://enthought.com/~cgalvan/eggs/pyOpenSSL-0.8-py2.5-win32.egg

[zooko]

Thanks, Chris! Hm, too bad ​http://enthought.com is not loading at the moment.

If you get a chance, could you upload that egg to ​http://testgrid.allmydata.org:8123 and paste the resulting URL into ​http://allmydata.org/trac/tahoe/wiki/Dependencies ? (According to setup.cfg​, this should make it automatically downloaded during builds -- right?)

Also, would you please build eggs for python 2.4 as well?

Thanks again!

[zooko]

Okay, I uploaded the pyopenssl egg to allmydata.org, but it turns out the egg doesn't work for me:

$ PYTHONPATH=./pyOpenSSL-0.8-py2.5-win32.egg python -c "import OpenSSL" 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Python25\lib\site-packages\PIL\__init__.py", line 11, in <module>
    
  File "c:\playground\allmydata\tahoe\installtahoe\trunk-hashedformat\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\SSL.py", line 7, in <module>
  File "c:\playground\allmydata\tahoe\installtahoe\trunk-hashedformat\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\SSL.py", line 6, in __bootstrap__
ImportError: DLL load failed with error code 182

[zooko]

Hm, I just tried it again after removing some other pyOpenSSL packages out of my C:\Python25\Lib\site-packages and this time I get a different error message:

$ PYTHONPATH=./pyOpenSSL-0.8-py2.5-win32.egg python -c "import OpenSSL"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Python25\lib\site-packages\PIL\__init__.py", line 11, in <module>
    
  File "C:\cygwin\tmp\x\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 7, in <module>
  File "C:\cygwin\tmp\x\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 6, in __bootstrap__
ImportError: DLL load failed: The specified module could not be found.

It's strange that the error message changed -- I had hoped that setuptools was arranging to use only the contents of this package.

[zooko]

So, if I unzip Chris's egg then I can "import OpenSSL" and it works (actually it fails with permissions errors, and then I "chmod +x *.pyd *.dll", and then it works). But if I rezip that directory (including the chmod +x) back into an egg and load it with PYTHONPATH=./the.egg python -c "import OpenSSL" then I still get this same error: DLL load failed.

See that weirdness with PIL in the stack trace? I removed the "PIL.pth" file I found in my system directory and that weirdness went away. Here's a cleaner stack trace with no PIL, of my attempt to load OpenSSL from the egg which results from me rezipping Chris's egg after chmod +x.

$ PYTHONPATH=./pyOpenSSL-0.8-py2.5-win32.egg python -c "import OpenSSL"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\cygwin\tmp\x\new\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\__init__.py", line 11, in <module>
  File "C:\cygwin\tmp\x\new\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 7, in <module>
  File "C:\cygwin\tmp\x\new\pyOpenSSL-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 6, in __bootstrap__
ImportError: DLL load failed: The specified module could not be found.

[zooko]

It fails likewise after being easy_install'ed:

$ easy_install.exe  ./pyOpenSSL-0.8-py2.5-win32.egg 
Processing pyOpenSSL-0.8-py2.5-win32.egg
Copying pyOpenSSL-0.8-py2.5-win32.egg to c:\python25\lib\site-packages
Adding pyOpenSSL 0.8 to easy-install.pth file

Installed c:\python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg
Processing dependencies for pyOpenSSL==0.8
Finished processing dependencies for pyOpenSSL==0.8

Zooko Brillnonywonx@your-fa38fa253f /tmp/x/tmp
$ python -c "import OpenSSL"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg\OpenSSL\__init__.py", line 11, in <module>
  File "c:\Python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 7, in <module>
  File "c:\Python25\lib\site-packages\pyopenssl-0.8-py2.5-win32.egg\OpenSSL\rand.py", line 6, in __bootstrap__
ImportError: DLL load failed: The specified module could not be found.

[cgalvan]

Hm these are some odd errors, I didn't run into any of these when testing on my machine :/ I will see what I can do about this later today when I get back from UNI.

[zooko]

Thanks for your help. I hope we can get this running smoothly as soon as possible, and then the imminent Tahoe-1.3.0 release can use it. I've been testing a bunch of other setup/build/install/test/doc improvements on my Windows machine (including your setuptools_trial plugin), but the lack of pyOpenSSL is currently blocking my testing.

[zooko]

I can build openssl and pyOpenSSL myself now, using mingw, by applying a patch to openssl-0.9.8i, but the resulting .egg has the same problem, as described over on ​https://bugs.launchpad.net/pyopenssl/+bug/238658

[cgalvan]

Here are the links to the Python 2.4 and 2.5 eggs of pyOpenSSL, respectively:

Python 2.4: ​http://testgrid.allmydata.org:8123/uri/URI%3ACHK%3Ao2cfflkjvhonp7ppfr4xwqnnya%3A4sw5j5r6xsebv7ex42fiuvyqm3punhb3hv7ayh3cip2y65c754sq%3A3%3A10%3A630020

Python 2.5: ​http://testgrid.allmydata.org:8123/uri/URI%3ACHK%3Avwfpw3sqs7wf7kganvibihx6ru%3Alqmbp7s2jjyqoeuxiq4du5kiae2v5kvssfdlycxqpaxpkclo2hsa%3A3%3A10%3A652359

[zooko]

Chris: will you please build a Python 2.6 egg? :-)

[zooko]

I built and uploaded a py2.6 egg:

​http://testgrid.allmydata.org:3567/uri/URI%3ADIR2-RO%3Asnrfwfxatrci35zdgjnzxxx2ke%3Aunarxv347edtku3xzmefy4mcdmfngxzeb72iyqcadbjzjpczjx5a/index.html

I think we can close this ticket.

[zooko]

This was fixed for 1.3.0.

[zooko]

It looks like at least the Python 2.6 egg is built wrong: #664 (Unable to start.)

[zooko]

JP Calderone has built binaries and attached them to the launchpad ticket: ​https://bugs.launchpad.net/pyopenssl/+bug/238658 . We should test them out and if they work upload them to our web site which serves up Tahoe-LAFS dependencies: ​http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs .

[zooko]

This ticket is just waiting for someone to manually test it (or better yet, program a buildbot to automatically test it). "It" in this case is that that binary .eggs that JP Calderone uploaded to the Launchpad ticket work.

[zooko]

cgalvan: are you planning to do this anytime soon? If you can confirm that the binary eggs that JP Calderone built work, then please close this ticket as "fixed". If you're not planning to do it, in say the next week then please assign this ticket to "nobody" in case someone else wants to pick it up before you get a round to it. Thanks!

[cgalvan]

Sorry for the delay on this one, I tested both eggs out today and I was able to run some basic scripts that use pyOpenSSL so I think it is safe to close this ticket.

[zooko]

Thanks, Chris!