[tahoe-dev] Tahoe Performance

[Brian Warner]

Wassim Drira wrote:
> Hi all,
> 
> I had done some tests on tahoe (I used 2 VMs on my machine  CPU:P9600 @
> 2.53GHz RAM:4GB ) and I found that uploading files of : 1kb takes 1.2s,
> 10kb takes 1.2s, 100kb takes 1.3s, 512kb takes 1.6s, .. (script and log
> are in attachment).

This is good data.. thanks for doing this work!

One piece of benchmarking advice: because Tahoe uses convergent
encryption, you should avoid uploading the same time multiple times (the
second time will be faster than the first, which will skew your
results). Append a counter or a few random bytes to the file each time
to avoid the convergence.

Also, it would be illustrative to identify the difference between
latency you're seeing and the throughput you might achieve. There is a
certain amount of constant overhead to each upload, and there are a
certain number of roundtrips. These roundtrips will increase the latency
for an upload, but no CPU or bandwidth is being used during that time,
so you could be uploading multiple files in parallel. This might not let
you store a single file in less than 1.2s, but you can probably store
ten files in 1.2s, which may be enough to achieve your goals.

I'd suggest a test framework which uses multiple simultaneous "tahoe
put" commands, each in a separate process (ideally on a different
machine than the Tahoe node, to factor out contention for CPU and local
disk between your test harness and the Tahoe node).

> In my application I need to store multiple small files (~10kb) per second.
> 
> Is there any way to maximize processing performance of small files in
> Tahoe (I need 0.2s/file) by disabling security, chunking, etc.

Disabling security is not likely to win you much: hashing and encryption
typically run at 10-50 mega*bytes* per second, so for small files it's
probably negligible.

If you have an easy way to aggregate small files into a larger one
before giving them to Tahoe (a tarball or zip file), that should
certainly help matters. It breaks the model a bit: you'd need to
remember both the Tahoe filecap plus the in-tarball name for the small
file, and you couldn't share just a single file with someone else
without also exposing everything else in that tarball to them.

The first big latency problem to tackle is the HTTP roundtrip between
your clients (like the "tahoe put" CLI command) and the tahoe node: it
requires an HTTP PUT and response cycle, process switching time, etc. To
fix this, you'll need to move your client code into the tahoe node. That
means adding a new twisted.application.service.Service child to the
tahoe Client instance, giving it methods that use the Uploader to
perform uploads, and possibly enhancing the webapi to let you instruct
your new code to do things (like uploading all files from a single
directory).

The quickest way to measure the HTTP roundtrip latency is to do "time
echo -n '' |tahoe put". That uploads a zero-byte file, which Tahoe does
by creating a "URI:LIT" filecap, which is not distributed and doesn't
even talk to the storage servers. Then compare the time it takes to do
that against uploading a 100-byte file, which *is* distributed and does
the usual tahoe upload work. On my laptop, the zero-byte file takes
971ms to upload, and a 1kB file takes about 1806ms. So I think that
about half of my latency is merely the context switching and
HTTP-through-localhost overhead of that local connection (and webapi
processing time).

Moving client code into the Tahoe node would be a lot easier if we had
some sort of plugin framework in place.

There are some other places where we could improve performance without a
huge amount of work or redesign. The uploader currently asks the storage
servers to hold shares in series, not in parallel, so the round-trip
time is magnified (if you use 10 servers, and it takes 50ms to ask each
one, it will take 500ms to ask all of them). With a bit of shuffling, we
could ask all 10 in parallel, and probably get the job done in just 50ms.

The upload protocol is a bit round-trip happy. There are a lot of small
pieces of data that get stored with each share, and we currently send
each piece with a separate network message. So we might use 7 roundtrips
for a small file. There is code to pipeline these requests, so it will
perform many of those roundtrips in parallel, but it would be better if
the protocol could aggregate/batch lots of small messages into a single
big one.


Anyways, hope that gives you some ideas to work with.

cheers,
 -Brian