alleged bug in SHA256 on win32 MSVC 2005 release build

[zooko]

On 2010-12-18, BlackEye <blackeye1@…> wrote:

I've been struggling with getting incorrect results from the SHA256 in CryptoPP 5.6.1.  I've only seen it happen in release builds with optimizations turned on.  If I build in debug mode, or turn off optimizations, I get the expected results.  I can also influence the output of correct or incorrect results in release mode by placing unrelated code around the calls to the hashing function.  For instance, the following piece of code produces the same hash in release mode, even though the data differs.  However, if I compile in debug mode or turn off optimizations I get two different hashes.  What can I do to remedy this?

#include <cryptopp/sha.h>
#include <iostream>

template <size_t nBytes, typename T>
T* alignup(T* p)
{
       union
       {
               T* ptr;
               size_t n;
       } u;
       u.ptr = p;
       u.n = (u.n + (nBytes-1)) & ~(nBytes-1);
       return u.ptr;
}

int main(int argc, char *argv)
{
       unsigned char digestbuff[32+16];
       unsigned char databuff[64+16];
       unsigned char *digest=alignup<16>(digestbuff);
       unsigned char *data=alignup<16>(databuff);

       for(int i=0; i<64; i++)
       {
               data[i]=0;
       }

       CryptoPP::SHA256::Transform((CryptoPP::word32 *)digest,(CryptoPP::word32 *)data);
       std::cout << digest[0] << " " << digest[1] << std::endl;

       data[0]=1;
       CryptoPP::SHA256::Transform((CryptoPP::word32 *)digest,(CryptoPP::word32 *)data);
       std::cout << digest[0] << " " << digest[1] << std::endl;

       return 0;
}

[zooko]

There are also reports on the Gentoo bug tracker of there being problems with SHA-256, but so far I haven't seen how to reproduce these issues or exactly what versions of binutils are involved:

• ​http://bugs.gentoo.org/show_bug.cgi?id=322713#c19
• ​http://bugs.gentoo.org/show_bug.cgi?id=343169
• ​http://bugs.gentoo.org/show_bug.cgi?id=343373

[zooko]

David-Sarah opened ​http://sourceforge.net/apps/trac/cryptopp/ticket/7 to track this issue in Crypto++. David-Sarah suspects that it might be a bug in the Microsoft compiler. I wonder if it could be another instance of ​http://sourceforge.net/apps/trac/cryptopp/ticket/6 . I'm not too worried about it since we haven't detected bad sha-256 results on any of our buildslaves (including win32 buildslaves) or any reports of such from users.

[davidsarah]

Replying to zooko:

David-Sarah opened ​http://sourceforge.net/apps/trac/cryptopp/ticket/7 to track this issue in Crypto++. David-Sarah suspects that it might be a bug in the Microsoft compiler. I wonder if it could be another instance of ​http://sourceforge.net/apps/trac/cryptopp/ticket/6 .

Unlikely for the reasons given in ​https://sourceforge.net/apps/trac/cryptopp/ticket/7#comment:6 .

When using Microsoft compilers, I usually enable optimizations only for specific source files where performance has a significant effect on overall execution time.

[davidsarah]

Replying to zooko:

There are also reports on the Gentoo bug tracker of there being problems with SHA-256, but so far I haven't seen how to reproduce these issues or exactly what versions of binutils are involved:

• ​http://bugs.gentoo.org/show_bug.cgi?id=322713#c19
• ​http://bugs.gentoo.org/show_bug.cgi?id=343169
• ​http://bugs.gentoo.org/show_bug.cgi?id=343373

These three are almost certainly the same bug as ​http://sourceforge.net/apps/trac/cryptopp/ticket/6 . They all involve the SHA-1 test going into an unbounded recursion.

[zooko]

Is there anything we should do for pycryptopp specifically as opposed to doing something about this for Crypto++? (See ​https://sourceforge.net/apps/trac/cryptopp/ticket/7 for this issue within the context of Crypto++.)

Perhaps we should add something to our setup.py​ to force optimization off when building with MSVC? Or add the pragma that David-Sarah suggested in ​https://sourceforge.net/apps/trac/cryptopp/ticket/7#comment:9 ?

I guess the next step would be to write a unit test that deterministically exercises this bug. This requires a buildslave with MSVC installed. I think Dcoder's buildslave would do:

​http://tahoe-lafs.org/buildbot-pycryptopp/builders/Dcoder%20Win7-64%20py2.6

[davidsarah]

Replying to zooko:

I guess the next step would be to write a unit test that deterministically exercises this bug.

If it is a misoptimization of C++ code, it will be difficult or impossible to provoke it from a Python unit test, unless the pycryptopp wrapper code happens to be misoptimized in this way (and if it were, we would probably have seen that already).

However, it should certainly be possible to write a Crypto++ unit test for this. That would be an issue for ​https://sourceforge.net/apps/trac/cryptopp/ticket/7 .

[zooko]

Hm, I'll bet this bug happens only with a specific version of the MSVC compiler, and that Python (and hence all Python modules such as pycryptopp) require a different specific version. Official CPython 2.6 and 2.7 for Windows is built, I think, with Visual Studio 2008. Presumably that compiler doesn't have this bug, which explains why we never see it on our buildbots or hear reports of it from our users.

Our MSVC buildbot ​https://tahoe-lafs.org/buildbot-pycryptopp/builders/Dcoder%20Win7-64%20py2.6/builds/19/steps/show-tool-versions/logs/stdio reports its cl version as:

cl: Microsoft (R) C/C++ Optimizing Compiler Version 15.00.30729.01 for x64 Copyright (C) Microsoft Corporation.  All rights reserved.  

According to ​http://msdn.microsoft.com/en-us/library/bb384838%28v=VS.90%29.aspx that indicates that it is Visual Studio 2008.

Closing this ticket as invalid.