end-to-end encoding self-test

[zooko]

Figure out what is going on with Ben Laurie's Tahoe node on his FreeBSD-6 using the public Test Grid, make it reproducible if possible in the unit tests, fix it.

[zooko]

Milestone 1.0.1 deleted

[zooko]

See tahoe-dev:

​http://allmydata.org/pipermail/tahoe-dev/2008-May/000542.html

[warner]

I'll read over the thread more closely and see if I can reproduce the problem locally.

I had first thought that it might just be the cap being corrupted, but that doesn't seem likely, given that the storage index is derived from the read-key.

[warner]

ah, ok, pycryptopp vs the stack. nevermind.

[zooko]

I'm working on adding "end to end" decryption, decoding, and parsing checks, i.e. in src/allmydata/test/test_encode.py​ upload some known-good, fixed ciphertext instead of encrypting, and likewise for SSK's and dirnodes.

[zooko]

The next step is to see what tahoe on Zandr's lenny-armel box does. The pycryptopp on that box seems to be giving incorrect decryption. Do the unit tests of tahoe on that box detect this misbehavior from pycryptopp?

[zooko]

Assigning this to Zandr in case he wants to look at it before the next time I circle around and log into his Linksys NAS box...

The hope is that when tahoe is used with a broken pycryptopp that gives wrong answers, that the tahoe unit tests fail.

[zooko]

Some tests on Zandr's lenny-armel box do indeed fail, such as allmydata.test.test_system.SystemTest.test_upload_and_download_random_key. See the buildbot results: ​http://allmydata.org/buildbot/builders/Zandr%20Lenny-armel/builds/37/steps/test/logs/stdio . However, these are failing because encrypting a string and then decrypting the ciphertext doesn't yield the original string. A subtler bug (and one that we've had before in pycryptopp) is when encrypting the string and decrypting the ciphertext does yield the original string, but only when you use the same (buggy) code for both encryption and decryption. This ticket is to add a "test vector", for example a fixed ciphertext, computed with a known-good implementation of AES and then hardcoded into the test file as a string literal, and assert that when you decrypt that fixed ciphertext you get the correct plaintext.

[zooko]

I'm making some progress on this, but it doesn't need to be done before Tahoe-LAFS v1.6.

[davidsarah]

Our tests are in general quite comprehensive, but this is a glaring omission.

[zooko]

I think we might have some tests to detect this now. Here's how to find out: mock out the AES class from pycryptopp (which is used in the immutable upload, mutable publish, immutable download, and mutable retrieve).

[zooko]

…and then set the mock to encrypt-decrypt wrongly but consistently and then see if any tests fail. Oh, and I guess this is also half of the job of writing the tests that we need! (If they aren't already there.) Shouldn't be too hard...

[zooko]

I'm demoting this from critical to major, since there are other urgent known-bugs (mostly in mutables) that are marked critical and this ticket, while certainly important, hasn't been touched in years.