[tahoe-dev] Crypto++ stable release

[Zooko O'Whielacronx]

On Mon, Jun 28, 2010 at 7:05 PM, Wei Dai <weidai at weidai.com> wrote:
>
> Jeff Walton had sent me a patch to fix Crypto++ crashing when loading
> Crypto++ more than once using TLD_GLOBAL. I didn't use his patch directly
> but made a change in SVN that should have an equivalent effect. However
> neither of us knows whether it would really fix the problem. If you have
> some time and idea how to test that, that would be great.

This is the infamous old pycryptopp ticket #9:

http://tahoe-lafs.org/trac/pycryptopp/ticket/9

We "solved" it for pycryptopp by setting RTLD_GLOBAL, which means (if
I understand C++ linking semantics correctly, which I don't and
apparently very few people do) that while the pycryptopp Python module
works fine now, if someone were to load a *different* Python module
which also dynamically linked to the Crypto++ dynamic library into a
Python interpreter along with pycryptopp then undefined behavior (such
as a crash or an exploitable vulnerability) would result.

To test your fix, we need to write a test of pycryptopp which loads
the pycryptopp Python extension module (which is itself a dynamically
linked library), which depends on the Crypto++ dynamically linked
library, and then also load another Python extension module which also
depends on the Crypto++ dynamically linked library. If my
understanding of the situation is correct, then this test will
sometimes trigger crashes on some platforms.

Then we can try upgrading to your latest Crypto++ with this patch in
it, and probably we also need to remove the code that sets the
RTLD_GLOBAL flag in:

http://tahoe-lafs.org/trac/pycryptopp/browser/trunk/pycryptopp/__init__.py?rev=683

If that fixes the crashes on those platforms then we'll call this an
improvement and release it. :-)

> Another thing is I fixed several bugs in the SHA-256 x86/x64 assembly code,
> so if you have any test code that exercises SHA-256, please run them. These
> are the SHA-256 bugs that have been fixed in SVN:
>
>         - incorrect hash on non-SSE2 x86 machines on non-aligned input
>         - incorrect hash on x86 machines when input crosses 0x80000000
>         - incorrect hash on x64 when compiled with GCC with optimizations
> enabled

Will do. I will commit a copy of your latest version of Crypto++
(excluding some source files that pycryptopp doesn't need) into here:

http://tahoe-lafs.org/trac/pycryptopp/browser/trunk/cryptopp

That will trigger these tests:

http://tahoe-lafs.org/trac/pycryptopp/browser/trunk/pycryptopp/test/test_sha256.py

to be run on all of our buildslaves:

http://tahoe-lafs.org/buildbot-pycryptopp/waterfall

By the way, François Deppierraz has previously reported a segfault in
SHA-256 on amd64:

http://tahoe-lafs.org/trac/pycryptopp/ticket/34

I don't know if he was able to reproduce it. François: can you
reproduce it? Wei Dai: does that stack trace look like it was an
instance of one of these SHA-256 bugs?

One final note: I would like to add a "quick start-up self test" to
pycryptopp which exercises the SHA-256 code at a minimal cost in time.
This would be analogous to the "quick start-up self test" that I added
to pycryptopp for AES:

http://tahoe-lafs.org/trac/pycryptopp/changeset/683/trunk

I guess for the SHA-256 test I would just copy this one unit test
which exercises different chunk sizes:

http://tahoe-lafs.org/trac/pycryptopp/browser/trunk/pycryptopp/test/test_sha256.py?rev=692#L87

I guess that would *not* (probably) exercise the really weird cases
such as the input crossing 0x80000000, but it would give me some
confidence. Users of pycryptopp would be encouraged to execute the
quick startup self test one time after loading pycryptopp. Of course
the biggest users of pycryptopp is Tahoe-LAFS and we would do that.

Regards,

Zooko

http://tahoe-lafs.org/trac/pycryptopp/ticket/43# add a quick start-up
self test for SHA-256
http://tahoe-lafs.org/trac/pycryptopp/ticket/44#
http://tahoe-lafs.org/trac/pycryptopp/ticket/44
http://tahoe-lafs.org/trac/pycryptopp/ticket/45# upgrade to latest
Crypto++ with SHA-256 bugfixes