pycryptopp fails self-tests on ARMv5 architecture

[zooko]

It looks like Crypto++ is writing into memory one byte earlier than it is supposed to -- the last byte of the buffer doesn't get written to, and all the other bytes get the value which the next byte over (address one higher) was supposed to get. Of course this could be a bug in the memory management of pycryptopp itself, but I didn't see one when examining the pycryptopp code, and this bug doesn't manifest on any of the other platforms pycryptopp has been run on, only on this ARMv5. Next steps are to check whether it is actually writing the first byte into the memory address one less than the beginning of the buffer, and to make a minimal test case of pycryptopp's memory management which doesn't actually invoke Crypto++ at all.

Oh, note that this test writes into two different buffers -- the first one is 15 bytes long and the second one is 17 bytes long. The first one is handled correctly, and the second one is written into the off-by-one wrong location. So it does seem like some code somewhere might be incorrectly padding, so that for example Crypto++ might think that it finished the first 16 bytes of CTR mode when it actually did only 15 bytes. That could explain it. Ah -- maybe I should check whether the first call to Crypto++ is incorrectly writing the 16th byte!

======================================================================
FAIL: test_encrypt_zeroes_in_two_parts (pycryptopp.test.test_aes.AES256)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/admin/pycryptopp/pycryptopp/test/test_aes.py", line 47, in test_encrypt_zeroes_in_two_parts
    self.failUnlessEqual(self.enc0, b2a_hex(ct1+ct2))
AssertionError: 'dc95c078a2408989ad48a21492842087530f8afbc74536b9a963b4f1c4cb738b' != 'dc95c078a2408989ad48a214928420530f8afbc74536b9a963b4f1c4cb738bad'

----------------------------------------------------------------------
Ran 30 tests in 50.729s

FAILED (failures=1
sh-3.2$ g++ --version
g++ (Debian 4.3.2-1.1) 4.3.2
Copyright (C) 2008 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE

[zooko]

Note that Crypto++ passes its own self-tests on this platform, and that it *does* test whether CTR mode correctly keeps track of successive non-block-size buffers. So now that I'm reminded of that I'm back to being more suspicious of pycryptopp or of Python.h than of Crypto++.

[zooko]

As another idea, it could be that the ARM architecture (with its gcc) is requires some alignment that the other architectures (with their compilers) don't.

[bhyde]

See also these emails that provide a probably explaination and work around.

​http://allmydata.org/pipermail/tahoe-dev/2009-June/001926.html

​http://allmydata.org/pipermail/tahoe-dev/2009-June/001927.html

"crypto++-5.6.0 on ARM ... works ... (upgrade your ARM boxes to
sid, or backport 5.6.0 to lenny"

[zooko]

This was fixed by upgrading to Crypto++ v5.6.0 in the embedded cryptopp subdirectory. (If you are linking to an external system Crypto++ then you are on your own with regard to this bug -- you're safe if your Crypto++ is >= v5.6.0 or if your architecture straightens out misaligned loads and stores for you.)

The patches that upgraded the embedded Crypto++ from v5.5.2 to v5.6.0 are [20090524005857-92b7f-7bfc567367206d5c933acb17ae09ae8f01b5174c], [20090526022036-92b7f-f8420997dee8bcee04acd5bff846aeb3b3fc4a40], [20090526023330-92b7f-7c88d93756bdd6a2666a8bfadb6886f8eee1f34b], [20090529012945-92b7f-d3e352f49d725fff288f23aac49fc668f6511d57], and [20090529020502-92b7f-7ddb7045d7283b0eed8abe287d7cf9e1c83507e3].