link against existing (system) libcrypto++.so

[warner]

The pycryptopp build process should make it possible to link against an
existing libcrypto++.so instead of always compiling its own. Reasons to do
this:

• faster: Crypto++ takes about 15 minutes to compile on our edgy buildslave, and this happens on every Tahoe build
• smaller memory footprint: if all the various python-to-C glue libraries (rsa.so, aes.so, etc) use their own copy of libcrypto++.so, an application that uses them will wind up with multiple copies in RAM.
• take advantage of newer versions: the system version may be more recent or faster than the one shipped with any given release of pycryptopp

Reasons to not do this:

• vulnerable to bugs in the system version: the system version of Crypto++ might have bugs that were fixed in the version shipped with pycryptopp
• static linking: if pycryptopp were to link statically against the pieces of Crypto++ that it needs, the resulting pycryptopp.egg might be transportable to a different machine. Note that currently it does *not* appear to do this: using 'ldd' on pycryptopp's rsa.so shows dynamic links to libcrypto++.so .

[ruben]

I'm working on packinging pycryptopp for Fedora, so it happens that I have to link against the system library libcryptopp.

I have attached a simple patch which does this, however I'm bumping into a problem while running the testsuite:

test_init_type_check (pycryptopp.test.test_aes.AES128) ...
terminate called after throwing an instance of 'CryptoPP::InvalidKeyLength?'

what(): AES/CTR: 1 is not a valid key length

Not sure what's going on, any help is appreciated.

[zooko]

I committed a patch -- [576] -- which adds an option --disable-embedded-cryptopp, which forces setup.py to use the system Crypto++ instead of the bundled source code of Crypto++.

Using it, I get the same kind of problem that you got, e.g.:

test_init_type_check (pycryptopp.test.test_aes.AES128) ... terminate called after throwing an instance of 'CryptoPP::InvalidKeyLength'
  what():  AES/CTR: 1 is not a valid key length
Aborted

and

test_hexdigest (pycryptopp.test.test_sha256.SHA256) ... terminate called after throwing an instance of 'CryptoPP::NameValuePairs::ValueTypeMismatch'
  what():  NameValuePairs: type mismatch for 'Separator', stored 'N8CryptoPP23ConstByteArrayParameterE', trying to retrieve 'N8CryptoPP23ConstByteArrayParameterE'
Aborted

I know what causes this, but not exactly how to fix it. I'll post more in a message to tahoe-dev.

[zooko]

​http://allmydata.org/pipermail/tahoe-dev/2008-October/000836.html

[warner]

stop using symbol comparison in AES.init exception catching

[warner]

I've attached a patch (aes-exception.diff) to stop using symbol comparison in the AES constructor's exception-handling code, along with an update to the test code. The new code catches all exceptions (not just CryptoPP::InvalidKeyLength) and turns them into an AES.Error python exception, with a stringified copy of the c++ exception. Works For Me.

An improvement might be to have the code examine the c++ exception to decide (with string comparison, rather than symbol comparison) whether we've got an InvalidKeyLength or something else, and then customize the python-side error message accordingly. I don't know enough c++ to do this easily.

[zooko]

Thank you for the patch!

If possible, I would prefer to use the structured C++ type information instead of regex matching in strings to detect the kind of an exception. This might be more important for the ecdsa key exceptions than for AES key exceptions. (All possible AES keys are valid as long as they are the right number of bytes in length, but ECDSA keys have mathematical structure.)

Here is a tutorial on using gcc export maps to control which symbols are exported from your library. The pycryptopp shared library ought not to be exporting symbols which are logical singletons and which are exported by libcryptopp.so:

​http://accu.org/index.php/journals/1372

If we can't figure out how to do this the Right Way, then I will accept patches like yours which either determine an exceptions type by string inspection, or else which treat all exceptions as of one indistinguishable type, but I would prefer not to.

[warner]

in looking at the sha256 hexdigest() code, I think it would be easiest to just remove that method, and require callers to hexlify it themselves. I looked at two approaches:

1. modify pycryptopp's SHA256_hexdigest() to hexlify using python instead of crypto++: basically insert a call to "import binascii; return binascii.b2a_hex(digest)". However, doing both the import and the call-into-python from there was more than a 5 minute project.
2. interpose a python module between sha256module.cpp and the outside world, with an eye towards wrapping the SHA256 object in one that has a hexdigest() method defined in python. That was more than a 5 minute project too, because I couldn't rename and change enough things to produce a working _sha256 .so instead of sha256.so (to make room for sha256.py).

So, just delete it. Callers can use binascii.b2a_hex(h.digest()) easily enough.

[zooko]

Hm, on reflection, I'm not sure that pycryptopp currently does anything with an exception from Crypto++ except re-raise it as a Python exception.

I hate, in principle to limit pycryptopp and its users to a single undifferentiated exception of type "exception from underlying library", when that underlying library is actually providing a carefully typed and informative exception.

But, in practice, this might make no different to our current code. I'll keep this in mind when returning to pycryptopp hacking after the tahoe-1.3.0 release.

[zooko]

I agree about .hexdigest() -- I wouldn't mind removing it. I'm hoping that before I get around to deleting it, someone will fix pycryptopp to export only the right symbols and then the current code including .hexdigest() will start to work even when linked against the system libcryptopp.so, though...

[zooko]

It looks like there is a nice fix:

​http://allmydata.org/pipermail/tahoe-dev/2009-February/001153.html

The next step is to configure some builders to test this issue, by installing a Crypto++ shared library on those builders and passing the --disable-embedded-cryptopp flag to setup.py build.

[warner]

Ok, we've got a "gutsy-syslib" builder now testing --disable-embedded-cryptopp. I fixed one debian-specific issue. The builder is currently red.

[warner]

To use that RTLD_GLOBAL fix, I suspect we'll need to create a python wrapper library: aes.py will do from _aes import * or something, and then _aesmodule.so will contain all the C++ glue. That will give us the opportunity to mess around with dlopen flags before and after the _aes import, without obligating all users of this library to be aware of the problem.

I tried briefly to rearrange aesmodule.cpp to make this work, but I failed. I've done this successfully a long time ago, but it was both C (instead of C++) and using distutils (instead of setuptools).

[zooko]

I wrote to the cryptopp mailing list asking for help with this, saying that the RTLD_GLOBAL seems a bit fragile:

​http://groups.google.com/group/cryptopp-users/browse_thread/thread/eb815f228db50380

[zooko]

Wei Dai suggested that I ask around on other lists, and I re-discovered the existence of the Python cplusplus-sig list, and summarized the problems and the four known ways forward:

​http://mail.python.org/pipermail/cplusplus-sig/2009-May/014531.html

If Debian and Fedora representatives could look at this and advise me how to proceed I would appreciate it. Adding Cc: a set of people who hack on Debian and/or Ubuntu and/or Fedora and who have previously offered to help.

[zooko]

Possibly related: the mysterious failure of dynamic_cast<> to downcast a private key in ecdsamodule.cpp:

​http://groups.google.com/group/cryptopp-users/browse_thread/thread/c6e8d97b25733ee2

Okay, probably not related, but still interesting...

[zooko]

Hooray! Now pycryptopp is ready for inclusion in Debian and Fedora!

Fixed by [20090527235925-92b7f-c7b71c236b9ca5ed4083327793cdea27c3201b32], [20090528001548-92b7f-962955aa4bd6122250415514932018f0e0d7cc29], [20090529002849-92b7f-65e8c52e4ba5f75ee8117453e52f8591e02d302a], [20090529012945-92b7f-d3e352f49d725fff288f23aac49fc668f6511d57], [20090529020502-92b7f-7ddb7045d7283b0eed8abe287d7cf9e1c83507e3], [20090529031429-4233b-0a7df0e293d2b79f7154ca8850d4c835cb0f8cf4], [20090529050608-4233b-737cdd78a23b99416521a53539984f58f8d81418], [20090529051221-4233b-ae789e6e82a9d527f47889397cdeac933d36d4c6], [20090529053603-4233b-d0c5a33e95706913adac3e1171c6178b691c6a16], [20090529054424-4233b-23f7e0df5baeb1809914157fd2e47021117aee46], [20090529055005-4233b-9bfab87d0682b8d2b596aa552d430844de4c8d5e].