[tahoe-dev] Tahoe-LAFS v1.8.0 potentially delayed by performance issue

[Zooko O'Whielacronx]

Okay folks we just now had a breakthrough and figured out what is
going on with this performance issue.

It has to do with downloading large amounts of data. The new 1.8.0c2
downloader wins against the 1.7.1 downloader for short downloads (I've
tried a few, mostly less than 10 MB out of this 1.5 GB file), but
loses when I download 100 MB of the file. We ran 100MB downloads under
a profiler and found evidence of a computation hotspot in the new
spans.py code. See #1170 for details.

So here's what you should do:

1. Expect a new release candidate of Tahoe-LAFS v1.8.0 in the next few
days which fixes this issue.

2. Do your testing and benchmarking on small reads for now. We
currently believe that Tahoe-LAFS v1.8.0 performs better than v1.7.1
on immutable file download as long as you are downloading less than
about 10 MB. If you see evidence to the contrary, please let us know!

By the way, is anyone interested in how Tahoe-LAFS handles lots of
small reads? I am. The v1.8 downloader brings Tahoe-LAFS a small step
closer to being able to compete with "NoSQL Databases" like Cassandra.
That's because the v1.8 downloader makes it possible to have a
potentially large immutable file of fixed-length records and read them
in "random access" mode. For what it is worth, Kevan's MDMF work
(coming in Tahoe-LAFS v1.9) should move it another small step closer
by making it possible to efficiently make small modifications to
mutable files, too.

Regards,

Zooko

http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1170# new-downloader
performs badly when downloading a lot of data from a file