Schema enforcement for HTTP protocol contents

[itamarst]

Many data inputs to the HTTP protocol need stringent checks, e.g. write lengths need to be positive, offsets need to be non-negative, hashes and secrets need to be the correct length, and so on.

So we need some way to enforce this.

[itamarst]

CDDL seems to support the above constraints.

[itamarst]

This Rust library seems like the best candidate for an implementation of CDDL: ​https://github.com/anweiss/cddl/issues/98. (The issue is me asking "are the caveats in README actually valid", I suspect they are not valid at this point.)

[itamarst]

Python JSONSchema library is highly unlikely to ever support bytes, given its goals.

[itamarst]

Sounds like the caveats for the Rust CDDL library are probably not a problem, and the author is working towards a 1.0, so seems worth wrapping with Python for Tahoe-LAFS.

[itamarst]

Schema enforcement features we'd need for current HTTP spec:

1. Dict value types ("this key's value must be bytes").
2. Dict keys ("this dict must only have these keys").
3. List lengths? Probably optional, could replace with dicts anywhere that comes up.
4. Integers must be positive, or not-negative.
5. List types ("this must be a list of integers", or "this must be a list of dicts with this structure").
6. Bytes vs. Unicode strings.
7. String length (for either kind of string).
8. Support both CBOR and JSON? Tricky for bytes.

Looking at CDDL:

1. Yes.
2. Yes.
3. Yes.
4. Yes.
5. Yes.
6. Yes.
7. There's a size control (how many bytes is it) which is fine for bytes, but a bit strange for Unicode strings in the general non-ASCII case.
8. The way CDDL works for JSON is by just saying "no bytes supported." Which is... a problem I guess.

So CDDL seems plausible, but might require maintaining two versions of the schema to deal with bytes-on-JSON.

[itamarst]

The _CBOR_ advice for bytes-on-JSON is "base64", so conceivably a CDDL validator could deal with bytes that way. But this is non-normative. I guess I'll see what Rust CDDL does.

[itamarst]

The Rust CDDL library does what the RFC suggests, there's no automatic "maybe these strings are bytes." But perhaps the author would be amenable... ​https://github.com/anweiss/cddl/issues/101

[itamarst]

Work in progress Python wrapper of anweiss' CDDL library: ​https://gitlab.com/tahoe-lafs/pycddl

[GitHub <noreply@…>]

In 0c18dca/trunk:

Merge pull request #1190 from tahoe-lafs/3802-cddl

CDDL-based schema validation for the HTTP storage server

Fixes ticket:3802