AES-CTR: easy way to modify the counter for random-access decryption

[warner]

I'd like to improve Tahoe's download process to allow random-access decryption. To do this, I need to be able to tell the AES decryptor object to process data from arbitrary points in the keystream.

I think it's possible to do this with the current API, by creating a new AES instance every time the counter jumps, and passing it an iv= argument which is a string into which we've packed the current offset. But this is too hard, and the iv= argument accepts arbitrary-length strings, making it difficult to confirm that we're supposed to pass in a string whose length is the same as the AES block size.

I'd like to have an extra argument to process(), which will reset the counter value. This argument should take a positive number (an int or long). If this argument is not provided, the counter should use the normal self-incrementing value. So:

 a = AES(key)
 data1 = AES.process("abcde")
 data2 = AES.process("fghij")
 data3 = AES.process("klmno")

should produce the same "data1", "data2", and "data3" as:

 a = AES(key)
 data2 = AES.process("fghij", counter=5)
 data3 = AES.process("klmno")
 data1 = AES.process("abcde", counter=0)

[warner]

​tahoe#798 is the tahoe ticket about adding true random-access download, which is dependent upon this feature being completed.

​tahoe#266 is a ticket about improving Helper partially-uploaded performance by using this feature.

[warner]

A not-too-unreasonable hack to accomplish most of this is:

def AES(key, offset):
    offset_big = offset // 32
    offset_small = offset % 32
    iv = binascii.unhexlify("%032x" % offset_big)
    decryptor = AES(key, iv=iv)
    decryptor.process("\x00"*offset_small)
    return decryptor

That returns an AES object which is ready for use at the given offset. Any
seeks have to create a new AES object, but that's not too expensive (27us on
my laptop).

However, AES(key, iv="0"*LEN) for any LEN other than 16 should throw an
exception. Calling it with iv strings shorter than 14 bytes causes it to use
uninitialized memory as part of the IV, not good.

[warner]

Oops, of course I meant:

def AES(key, offset):
    offset_big = offset // 16
    offset_small = offset % 16
    iv = binascii.unhexlify("%032x" % offset_big)
    decryptor = AES(key, iv=iv)
    decryptor.process("\x00"*offset_small)
    return decryptor

A quick unit test reveals the problem with using 32 by mistake.

[zooko]

We should really extend pycryptopp to allow arbitrary starting counters.

[davidsarah]

Incidentally, ​EncryptedTemporaryFile in the Tahoe-LAFS SFTP frontend currently uses warner's hack from comment:3.

[weidai]

Crypto++'s StreamTransformation? interface (implemented by CTR mode and XSalsa20) has a Seek() function that allows random access encryption/decryption without having to change the IV. I'm not sure if you guys were aware of that.

[from_pycon]

Hi,

I played a bit with this. You can find my results in branch ticket_18 of https://git.gitorious.org/my-tahoe-lafs/my-tahoe-lafs.git.

​https://gitorious.org/my-tahoe-lafs/my-tahoe-lafs/commits/ticket18

[from_pycon]

Oh yeah, there are no unit tests on the decryption side (although both ciphers are symmetric, so it might be OK).
And I suspect there might be a couple of memleaks on the way I handle PyObjects (read, I never decrement their ref. counter).

[from_pycon]

OK. Please see branch ticket18_take2 of https://git.gitorious.org/my-tahoe-lafs/my-tahoe-lafs.git (yes, with no underscore between 'ticket' and '18').

The code should be tidier now.