cloud/S3 backend fails to redact ProductToken and UserToken from S3 error messages

[daira]

Here's an example of LeastAuthority.com secrets being leaked in an S3 error message (I've replaced the actual secrets with "THIS_SHOULD_NOT_BE_HERE" for this bug report):

[Failure instance: Traceback: <class 'lae_automation.endtoend.CheckFailed'>:
 Error for 107.22.17.1: could not create test file: [Failure instance: Traceback (failure with no frames):
 <class 'allmydata.mutable.common.NotEnoughServersError'>: ('Publish ran out of good servers, last failure was:
 [Failure instance: Traceback (failure with no frames): <class \'foolscap.tokens.RemoteException\'>:
 <RemoteException around \'[CopiedFailure instance: Traceback from remote host -- Traceback (most recent call last):\n
  File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/tcp.py", line 277, in connectionLost\n
    protocol.connectionLost(reason)\n
  File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/web/client.py", line 191, in connectionLost\n
    self.factory._disconnectedDeferred.callback(None)\n
  File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py", line 362, in callback\n
    self._startRunCallbacks(result)\n
  File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0-py2.6-linux-i686.egg/twisted/internet/defer.py", line 458, in _startRunC\n
\n-- TRACEBACK ELIDED --\n\n
  client/base.py", line 46, in error_wrapper\n
    raise fallback_error\n
allmydata.storage.backends.s3.s3_common.TahoeS3Error: (\'400\', \'400 Bad Request\',
\'<?xml version="1.0" encoding="UTF-8"?>\\n
<Error><Code>InvalidToken</Code>
<Message>The provided token is malformed or otherwise invalid.</Message>
<RequestId>266AB3D40D3E8F00</RequestId><HostId>IXcXMiM5tH07dLaANbZsgKe4rqkFF7yMBmfGlhWqZfdd9i6FqUiuUcsgEc6cmrAW</HostId>
<Token-1>{ProductToken} THIS_SHOULD_NOT_BE_HERE </Token-1>
<Token-0>{UserToken} THIS_SHOULD_NOT_BE_HERE </Token-0>
</Error>\')\n
]\'>\n]', None)

The storage server is running ticket999-S3-backend, but the problem also occurs for the current 1819-cloud-merge branch. (The latter redacts SignatureDoesNotMatch errors but not InvalidToken errors.)

[daira]

Note that the s3secret key is not leaked by this problem, so the impact is quite limited. However, the new end-to-end monitoring has a greater chance of revealing error messages to the public monitoring list.

[exarkun]

The established line of development on the "cloud backend" branch has been abandoned. This ticket is being closed as part of a batch-ticket cleanup for "cloud backend"-related tickets.

If this is a bug, it is probably genuinely no longer relevant. The "cloud backend" branch is too large and unwieldy to ever be merged into the main line of development (particularly now that the Python 3 porting effort is significantly underway).

If this is a feature, it may be relevant to some future efforts - if they are sufficiently similar to the "cloud backend" effort - but I am still closing it because there are no immediate plans for a new development effort in such a direction.

Tickets related to the "leasedb" are included in this set because the "leasedb" code is in the "cloud backend" branch and fairly well intertwined with the "cloud backend". If there is interest in lease implementation change at some future time then that effort will essentially have to be restarted as well.