#18 new enhancement

AES-CTR: easy way to modify the counter for random-access decryption

Reported by: warner Owned by:
Priority: major Milestone: 0.7.0
Version: 0.5.1 Keywords: review-needed
Cc: Launchpad Bug:


I'd like to improve Tahoe's download process to allow random-access decryption. To do this, I need to be able to tell the AES decryptor object to process data from arbitrary points in the keystream.

I think it's possible to do this with the current API, by creating a new AES instance every time the counter jumps, and passing it an iv= argument which is a string into which we've packed the current offset. But this is too hard, and the iv= argument accepts arbitrary-length strings, making it difficult to confirm that we're supposed to pass in a string whose length is the same as the AES block size.

I'd like to have an extra argument to process(), which will reset the counter value. This argument should take a positive number (an int or long). If this argument is not provided, the counter should use the normal self-incrementing value. So:

 a = AES(key)
 data1 = AES.process("abcde")
 data2 = AES.process("fghij")
 data3 = AES.process("klmno")

should produce the same "data1", "data2", and "data3" as:

 a = AES(key)
 data2 = AES.process("fghij", counter=5)
 data3 = AES.process("klmno")
 data1 = AES.process("abcde", counter=0)

Change History (11)

comment:1 Changed at 2009-08-27T06:33:09Z by warner

tahoe#798 is the tahoe ticket about adding true random-access download, which is dependent upon this feature being completed.

tahoe#266 is a ticket about improving Helper partially-uploaded performance by using this feature.

comment:2 Changed at 2010-02-02T07:18:11Z by warner

A not-too-unreasonable hack to accomplish most of this is:

def AES(key, offset):
    offset_big = offset // 32
    offset_small = offset % 32
    iv = binascii.unhexlify("%032x" % offset_big)
    decryptor = AES(key, iv=iv)
    return decryptor

That returns an AES object which is ready for use at the given offset. Any
seeks have to create a new AES object, but that's not too expensive (27us on
my laptop).

However, AES(key, iv="0"*LEN) for any LEN other than 16 should throw an
exception. Calling it with iv strings shorter than 14 bytes causes it to use
uninitialized memory as part of the IV, not good.

comment:3 Changed at 2010-02-02T07:34:16Z by warner

Oops, of course I meant:

def AES(key, offset):
    offset_big = offset // 16
    offset_small = offset % 16
    iv = binascii.unhexlify("%032x" % offset_big)
    decryptor = AES(key, iv=iv)
    return decryptor

A quick unit test reveals the problem with using 32 by mistake.

comment:4 Changed at 2010-07-06T05:12:37Z by zooko

We should really extend pycryptopp to allow arbitrary starting counters.

comment:5 Changed at 2010-07-06T20:14:03Z by davidsarah

Incidentally, EncryptedTemporaryFile in the Tahoe-LAFS SFTP frontend currently uses warner's hack from comment:3.

comment:6 Changed at 2010-08-19T09:10:56Z by weidai

Crypto++'s StreamTransformation interface (implemented by CTR mode and XSalsa20) has a Seek() function that takes a byte offset and allows random access encryption/decryption without having to change the IV. I'm not sure if you guys were aware of that.

Last edited at 2012-03-15T07:50:15Z by zooko (previous) (diff)

comment:7 Changed at 2012-03-14T00:56:11Z by from_pycon


I played a bit with this. You can find my results in branch ticket_18 of https://git.gitorious.org/my-tahoe-lafs/my-tahoe-lafs.git.


comment:8 Changed at 2012-03-14T01:03:24Z by from_pycon

Oh yeah, there are no unit tests on the decryption side (although both ciphers are symmetric, so it might be OK).
And I suspect there might be a couple of memleaks on the way I handle PyObjects (read, I never decrement their ref. counter).

Last edited at 2012-03-15T07:55:42Z by zooko (previous) (diff)

comment:9 Changed at 2012-03-16T10:06:04Z by from_pycon

OK. Please see branch ticket18_take2 of https://git.gitorious.org/my-tahoe-lafs/my-tahoe-lafs.git (yes, with no underscore between 'ticket' and '18').

The code should be tidier now.

comment:10 Changed at 2012-03-21T03:24:24Z by zooko

  • Keywords review-needed added

comment:11 Changed at 2013-01-30T17:33:46Z by zooko

  • Milestone set to 0.7.0
Note: See TracTickets for help on using tickets.