[tahoe-lafs-trac-stream] [tahoe-lafs] #510: use plain HTTP for storage server protocol

[tahoe-lafs]

#510: use plain HTTP for storage server protocol
------------------------------+---------------------------------
     Reporter:  warner        |      Owner:  zooko
         Type:  enhancement   |     Status:  new
     Priority:  major         |  Milestone:  2.0.0
    Component:  code-storage  |    Version:  1.2.0
   Resolution:                |   Keywords:  standards gsoc http
Launchpad Bug:                |
------------------------------+---------------------------------

Comment (by simeon):

 Replying to [comment:31 simeon]:
 > "...everything else done at a client end, apart from exchanging objects
 between backend nodes and deprecation."
 >
 > Perhaps since deprecation is handled by the server, then
 locking/blocking should also be

 Oh. And hashsumming of data, and making symlinks between a filename and a
 hashsum db blob. Your clients want to store data, so they want to know it
 was received intact, so you have to give them back a hashsum of the data.
 And often the data they want to upload already exists, so you can avoid
 the upload. But they want to use a different filename, so a symlink (or
 equivalent?).

 But hashsumming on the server leads to a potential DoS. A malicious client
 can force the server to hashsum a lot by uploading arbitrary data, putting
 a compute-load on it. I suppose a hashcash system can be used to reduce
 the ability of clients to do this?

 Perhaps this is an argument for finite-length blocks with the client being
 forced to at least provide hashsums as they are uploaded, but this adds
 complexity to the client and to the protocol, without really doing a lot
 to prevent DoS. Most clients are not CPU-bound, they are network-bound.

 If so, a more effective way would be to force the client to download the
 data back, and not store it if they won't.

 Combined with LRU deprecation these would reduce the ability of random
 clients to replace useful content with completely unuseful random crud.

 In any case, I don't see any better way for the system to work, than for
 the server to hashsum the data ASAP after receipt, and report the value to
 the client prior to considering a transaction 'complete'.

 Here is a picture of the pipeline I envisage

 data->compress?->hashsum1->encrypt+hashsum2?->store->hashsum2+decrypt?->hashsum1->decompress?->data.

 The compress stage is for two purposes, 1) to make the file small to
 decrease the cost of hashsum and encryption, and 2) to make the encryption
 as resilient as possible to deliberate attack (I am not a cryptographer,
 but it's my understanding that that's how things work). But the compress
 should probably only be done if it's useful. Either way, I guess leave
 that decision up to the client (which could be an intermediate transform
 web-application, rather than the endpoint, btw).

 The current question is, which parts of this transform should be specified
 in the URI?

 I think the URI must be capable of specifying the following (question-
 marks where I'm not sure the item is useful):

 * ? file size or size-class ?
 * ? checkdigit (a one-byte checksum of the hashsum, to detect user typos)
 ?
 * hash algorithm (if omitted, default to a pre-configured value)
 * truncatable hashsum (if omitted, query based on filename, if provided,
 and if allowed by config)
 * optional sequence number in case of hash collisions (not guaranteed to
 be consistent between nodes, or across time, server maintained, and when
 dups exist but one isn't specified, then return a list somehow to allow
 the client to choose)
 * optional filename

 Certain metadata relating to files would be stored, in other ordinary db
 files requested using the same scheme.

 Private metadata1

 * ? magic number ?
 * hashsum2->hashsum1
 * encryption parameters
 * compression algorithm

 Private metadata2

 * ? magic number ?
 * mapping hashsum1->hashsum2[,hashsum2,hashsum2...]

 btw I hate XML, so if you want to upset me, use XML for your metadata.
 I'll use something else for mine ;-)

 The metadata files present a problem for keeping the encryption secure. If
 the file is to be constantly changed, the differing versions with
 differing content, but a lot of unchanged content and the same key each
 time, provides a way for an observer to break the encryption. Or if the
 key is changed often, a different pitfall presents, the encryptor
 potentially spends a lot of entropy, so the key is not very strong. I
 think.

 So although it may seem optimal to aggregate metadata for many blobs into
 a single metadata file, perhaps not. Hard not to hit one of those two
 attack vectors, in this scenario. But if each file has a small metadata
 file, that small file could be encrypted using an expensive encryption
 algorithm, I guess. Not a cryptographer, so I could be very wrong.

 For private use, the metadata need not be uploaded to the server at all.
 The user would need to be aware of how that works, and not lose the
 metadata, of course. Perhaps some combination of local caching and
 infrequent upload of metadata to the cloud would work?

 Now a first guess (and now, later, a second one) is that to the backend,
 all of the information on which stages of the pipeline to apply, and what
 values to expect, is just cruft to be either ignored or stored without
 checking. That would make for a very simple and fast upload storage
 server.

 There is no point having the client tell the server what hashsum to
 expect, because there is no defense against malicious clients in this
 case. A hashsum can only be calculated once the data is received. Or is
 this an argument for smallish blocks? Make the client provide a hashsum
 every 100k or whatever, and stop talking if they get one wrong? It still
 doesn't help much to prevent DoS, just means the client spends the same
 effort as the server. A DDoS still works.

 Actually a malicious client would be best to upload spurious data WITH
 correct hashsum, so that the db fills up with rubbish. So there is no
 reason to worry overly much about checking hashsums on the server. Do or
 don't do, it doesn't affect the chance for a DoS. Am I right?

 You may as well just store whatever you receive, hashsum it, label it so
 it can be retrieved by a client who asks using the hashsum, and leave it
 to the client to notice if the upload failed, ie if the data did not end
 up with the expected hashsum (if they even had calculated what to expect:
 a client could be dumb and just trust the server to return the correct
 value).

 And this is why a LRU deprecation policy is needed. Because since there is
 no way for the server to know if the data offered will ever be used, it
 must know which data to drop if out-of-space. So drop oldest, least used
 data. More useful UNIX filesystem semantics: last-accessed, and archive-
 bit.

 One could even accept uploads blindly, not even checksum on the server,
 just have client report one, and assume client tells truth about checksum,
 later LRU deprecate, leave it to other clients to integrity check what
 they receive? I doubt this method would make for happy users, there may be
 many DoS being passed on to other users. Hmm.

 Perhaps modify by server verify hashsum only of files that survive the
 LRU? Or just have a lazy thread hashsumming in the background and deleting
 those that don't match whatever a client specified?

 So possibly in the API the POST needs to specify the expected hashsum. Or
 not. *TODO which is easiest, which is best?

 If you trust clients to indicate good/bad of files they get, it seems to
 me like just another thing they could lie about, just shifts the DoS. Same
 with flagging files that get many dropped downloads: a malicious client
 could make partial downloads on purpose to cause particular files to be
 flagged as bad. Storage space is cheap, and even cheaper in a distributed
 net (we hope!). These feedback/metrics systems might work with a
 signature/authority scheme, but then, same diff, you may as well put the
 trust earlier in the piece and flag bad uploads soon-as.

 The case of breaking up files I think is just a usage case where the
 client does that, and stores the results into the server. You could, if
 this is planned, optimise the server to work best for specific file sizes.
 And in this situation, you might choose to look at this client as a
 transform layer, below the transform layer that accepts data from the
 user, and riding on top of the base layer of the storage server API. I
 think the URI schema should be generic, and yet succinct enough to cater
 to the top (human user) layer, and lower-layer uses should be specific
 subsets of that syntax, or with certain parameters filled with
 robotically-derived content.

 Again, apologies for lengthy and perhaps unreadable text, too tired now to
 do anything except hit Submit Changes. ;-) Thanks for listening if you
 read this far, and I hope it's helpful!

-- 
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/510#comment:32>
tahoe-lafs <https://tahoe-lafs.org>
secure decentralized storage