when cryptography has random-access AES, update helper to use it

[warner]

to build the offloaded-uploader (#116), we need the ability to do AES CTR mode at arbitrary places in the input stream. I think a second (optional) argument to aes.AES.process() that accepts a byte offset would do the trick. I'm imagining something like:

   def process(self, data, offset=None):
       if offset is None:
           offset = self._last_offset
       block = aes_encrypt(self._key, number_as_string(offset))
       output = xor(data, block)
       self._last_offset = offset + len(data)
       return output

(with all the appropriate joy of handling block boundaries, of course)

[warner]

hmm, robk astutely points out that we might want to re-encrypt the data on the second pass anyways, to build up the hash trees. So we can build the offloaded uploader without this. Lowering the priority.

[warner]

Note: once pycryptopp can do this, revisit upload.py EncryptoAnUploadable.read_encrypted, to allow it to avoid encrypting all the data that we're going to throw away (when hash_only==True).

[warner]

it sounds like zooko just added this feature to pycryptopp-0.3.0, right?

[zooko]

Yes. You can pass an "iv" argument to the constructor of an AES object now.

[warner]

ok, changing the ticket description to reflect that this is now an action item for the helper, rather than pycryptopp

[warner]

hrm. Tell me about this "iv" argument: is it a string or an integer? I think I'd rather have pycryptopp be responsible for the pack-int-to-string and block-alignment issues.. it is more likely to get them right than I would.

It would also be nice to be able to pass an offset into process(), so you could create one AES object and then seek around with it (rather than creating a new AES object for every segment), but if that were a hassle I suppose I could live without it.

What happens if my segment starts on a non-AES-blocksize boundary?

Incidentally, I added ​pycryptopp#18 to request this feature, having forgotten about this #266 ticket.

[warner]

As I understand it, pycryptopp's iv= argument is *not* sufficient for this, at least not without a huge amount of work on the tahoe side. iv= is a string, which defaults to an AES-sized block of all zeros. To process data at an arbitrary location, the application must figure out the right offset, pack that value into the IV block, handle non-multiple-of-the-blocksize shifts, and then keep track of exactly how much data you pass into process() if you want to call it more than once. Then, to actually seek() to a different location, you have to throw away that AES object and make a new one, since the AES() constructor is the only place that takes an iv= argument, not process().

Updating title to reflect this. Note that pycryptopp ticket 18 is still the feature request: this ticket is to remind us (on tahoe) that this sort of feature is blocked on pycryptopp-18. Also note that better random-access downloading (i.e. anything other than starting at the beginning of the file and fetching everything from there to the point of interest) also requires random-access AES.

[exarkun]

ticket:3031 switched Tahoe-LAFS from pycryptopp to cryptography but cryptography also doesn't offer a random-access AES so I'm modifying this ticket to be about the same problem but with cryptography.

[exarkun]

I filed ticket:3230 for considering an alternate solution. From #cryptography-dev there was neither a clear yes or no on whether providing a simpler API in cryptography would be sensible. The use case is slightly obscure and the functionality is already available, just with a slightly inconvenient API. However, OpenStack? Swift does basically the same thing as Tahoe-LAFS: ​https://github.com/openstack/swift/blob/ae36030278b244148363d4a1c6ed0c7fc7cb5204/swift/common/middleware/crypto/crypto_utils.py#L76-L84

The fact that there are two applications seemed to carry some weight.