[tahoe-dev] bugs in SHA-256 Re: Crypto++ stable release

[Zooko O'Whielacronx]

Dear cryptopp-users, tahoe-dev, and Wei Dai:

Brian Warner found a new, reproducible, case of a bug in SHA-256:

http://tahoe-lafs.org/trac/pycryptopp/ticket/34#comment:3

We're hooking up that machine to be a buildslave so that we'll have an
automated case of it and a public record of it. That will probably
come on-line tomorrow. (it is named "Brian Ubuntu-amd64 linode" on the
pycryptopp buildbot [1].)

Also Brian suggested to use the Python standard library's hashlib [2]
instead of Crypto++'s implementation of SHA-256 and when I opined that
hashlib might be slower, he offered to benchmark them against one
another. hashlib was added in Python 2.5, and until now we've
supported Python 2.4, but I don't think I would mind very much if the
upcoming Tahoe-LAFS v1.8 no longer supported Python 2.4.

On the other hand I'm reluctant to use hashlib because I don't know
much about the quality-control that has been used on it. I know that
it has two implementations, one using OpenSSL and another copied from
LibTomCrypt. I've seen a couple of bugs in the SHA-256 implementation
that was in PyCrypto which was also copied from LibTomCrypt.

Anyway, I know a lot about the quality-control process for Crypto++
and pycryptopp, and would generally feel happier continuing to debug
and add tests for it than switch to something else.

Finally, we should note that all of the bugs in Crypto++'s SHA-256
that I can recall off the top of my head have been in the assembly
implementation. Another strategy we adopt is to turn off assembly and
use the C++ implementation. This would also simplify pycryptopp ticket
#37. I tend to think of that as a potentially useful stopgap measure,
but in fact I expect that we'll be able to track down and fix this bug
soon enough that we won't really need to deploy a no-assembly build.

Anyway, I'm writing to let Wei Dai know that I'm going to prioritize
some pycryptopp maintenance work this week, so hopefully this will
give good testing of the code which is a candidate for the next stable
release of Crypto++. Here are the things I would like to accomplish
ASAP (help would be greatly appreciated!):

 * http://tahoe-lafs.org/trac/pycryptopp/ticket/37# Win64 and MSVC9
compilation fixes
 * http://tahoe-lafs.org/trac/pycryptopp/ticket/43# add a quick
start-up self test for SHA-256
 * http://tahoe-lafs.org/trac/pycryptopp/ticket/44# test what happens
if another Python module loads Crypto++ dynamic link library
 * http://tahoe-lafs.org/trac/pycryptopp/ticket/45# upgrade to latest
Crypto++ with SHA-256 bugfixes
 * read potentially helpful explanation of the dynamic linking issues:
http://mail.python.org/pipermail/cplusplus-sig/2003-July/004594.html

Regards,

Zooko

[1] http://tahoe-lafs.org/buildbot-pycryptopp/waterfall
[2] http://docs.python.org/library/hashlib.html