Benchmarks for basic immutable Tahoe operations (upload and download)

[itamarst]

Motivation:

1. We don't want HTTP to be slower than Foolscap.
2. It's not clear how fast these operations are in general.

Thus: there should be a way to measure how long these operations take.

First pass will run over loopback, but latency can be added-on on Linux (e.g. ​https://bencane.com/2012/07/16/tc-adding-simulated-network-latency-to-your-linux-server/).

[itamarst]

For a small 2000 byte file with a single share uploaded to a single node, an upload is 3 sequential HTTP requests, and a download is 4 sequential HTTP requests. So with high latency this gets quite slow.

Next, I tested HTTP over loopback, but with RTT latency of 60ms (30ms one way) artificially added. Bandwidth is quite high, so not likely to be a bottleneck.

File size	Upload clock time	Upload CPU time	Download time	Download CPU time
2KB	300ms	30ms	300ms	65ms
2MB	700ms	70ms	3,200ms	680ms
20MB	3,500ms	600ms	31,000ms	6,200ms

This is all sharing a single CPU core for both client and server, it's one big process.

So at first glance:

• Immutable uploads use parallelism, and main bottleneck is latency. Upload time is not linear with size, . However, sequential uploads will have minimal time no matter what the bandwidth.
• Immutable downloads have a massive latency hit, probably due to not utilizing parallelism well, so large files suffer from latency hit even when bandwidth is not a bottleneck. Also using a lot of CPU.

I then tested Foolscap with 20MB file. Upload 3,200ms clock time, 500ms CPU time, so HTTP upload is worse but not too bad in comparison. Foolscap download was 20,800ms clock time and 2,100 CPU time. So HTTP download is much worse than Foolscap, but Foolscap downloads are still quite bad.

[itamarst]

Changing default segment size to be bigger helps a lot (mostly a workaround).

[itamarst]

The problems with increasing default segment size:

• Some code actually relies on it (e.g. ​https://github.com/tahoe-lafs/tahoe-lafs/blob/master/src/allmydata/immutable/downloader/node.py#L74) though _hopefully_ that's just an optimization. Would need testing of all relevant places that rely on it, at least.
• It doesn't help with existing downloads.