[tahoe-dev] linking pycryptopp against system provided cryptopp

[Ruben Kerkhof]

Hi Zooko,

On Oct 20, 2008, at 11:37 PM, zooko wrote:

> Greeting, Ruben Kerkhof.
>
> I'm excited to hear that you are considering packaging Tahoe for
> Fedora.  Fedora seems to be the most technically advanced Linux
> distribution nowadays (sorry if that hurts anyone's feelings...).

Thanks!

> Micah Anderson is considering packaging Tahoe for Debian, as well, so
> you and he might be able to cooperate on some of the issues that
> you've found.

That's an excellent idea, we might be able to share a few things, like  
initscripts for example.

>
> I have committed a patch, pycryptopp [579] (see below for URL), which
> adds a "--disable-embedded-cryptopp" option to setup.py build".
> However, when I use it on my Ubuntu Hardy workstation, I get errors
> like the ones that you reported, e.g.:
>
>> test_init_type_check (pycryptopp.test.test_aes.AES128) ...
> terminate called after throwing an instance of
> 'CryptoPP::InvalidKeyLength'
>>  what():  AES/CTR: 1 is not a valid key length
>> Aborted
>
> and
>
>> test_hexdigest (pycryptopp.test.test_sha256.SHA256) ... terminate
> called after throwing an instance of
> 'CryptoPP::NameValuePairs::ValueTypeMismatch'
>>  what():  NameValuePairs: type mismatch for 'Separator', stored
> 'N8CryptoPP23ConstByteArrayParameterE', trying to retrieve
> 'N8CryptoPP23ConstByteArrayParameterE'
>> Aborted
>
> I know what causes these, but I'm not precisely sure how to fix it.
>
> The problem is that in gcc 3.0 and newer, type equality is determined
> by address comparisons (i.e. "this_type_ptr == that_type_ptr") instead
> of string comparisons (i.e. "strcmp(this_type_str, that_type_str) ==
> 0").
>
> http://www.gnu.org/software/gcc/faq.html#dso
>
> This means that if a type gets used in more than one compilation unit,
> such as the "InvalidKeyLength" exception which is defined in
> cryptopp/cryptlib.h, and is raised in libcryptopp.so and caught in
> aes.so, and the type is compared across compilation units, then the
> linker and/or loader has to detect that the usages of this type both
> actually mean the same type and deduplicate the type symbol.
>
> (That's why in the second error message above, the C++ error says
> "type mismatch" and then lists two types which are the same string.)
>
> Now, when you compile pycryptopp with its own included copy of the
> Crypto++ source code, then both compilation modules are linked in the
> same linker step and this error doesn't occur.
>
> But when you compile pycryptopp against the system libcryptopp.so,
> then it does.  Based on that gcc FAQ entry above, I have two guesses
> as to how this could be fixed:

I barely understand this, c++ and linkers are not my strongest points,  
but let's see what we can find out :-)
>
> 1.  Maybe when the system libcryptopp.so is built, we need to make
> sure that it doesn't get built with -Bsymbolic.  This would be easy to
> test by building libcryptopp.so outside of the pycryptopp tree and
> linking against it.

Cryptopp itself is under review for Fedora as well at the moment:
https://bugzilla.redhat.com/462104
I just verified it doesn't get build with -Bsymbolic, but to be sure  
you did not make a typo above, does or doesn't it need to be build  
with -Bsymbolic?
I just played with both versions, and also compiled pycryptopp with - 
Wl,-Bsymbolic, and all combinations give the same errors.

> 2.  Maybe when python dlopens pycryptopp's aes.so, python needs to
> pass the RTLD_GLOBAL flag to dlopen(), and pycryptopp's aes.so and
> libcryptopp.so (both?) need to be built with -E.  Perhaps python
> already passes RTLD_GLOBAL -- I haven't checked.

Here's a rather old thread on python-dev that seems to describe  
exactly what we are noticing:
http://mail.python.org/pipermail/python-dev/2002-May/023923.html

> Okay, I hope this helps.  By the way, did you really post your patch
> to the pycryptopp trac four weeks ago?  I'm sorry I didn't notice
> until you posted to tahoe-dev.  I've configured the pycryptopp trac to
> send me e-mail whenever anyone opens a ticket from now on.
>
> Regards,
>
> Zooko

Yeah, I did :-)
I should have mailed the list much sooner though...

Kind regards,

Ruben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1638 bytes
Desc: not available
Url : http://allmydata.org/pipermail/tahoe-dev/attachments/20081021/7703e53b/attachment.bin