[tahoe-dev] 1.8.0 progress report

[Zooko O'Whielacronx]

Thanks to everyone who helped out this past weekend!

We made some progress on the remaining issues: documentation issues,
some edge cases in downloader which are probably exceedingly rare in
practice, and the vexing "is this a critical performance regression?"
issue (#1170).

http://tahoe-lafs.org/trac/tahoe-lafs/roadmap

Kyle's overnight benchmarks that he just published [1] give me some
assurance that 1.8.0c3 is typically at least 90% as fast as 1.7.1 even
in the worst case. Kyle's setup should be just about the worst case
for 1.8.0c3, since there is extremely low network latency and all of
the servers are almost exactly as fast as one another. In a case where
there is higher network latency (i.e. you are downloading over the
Internet, not a LAN), then 1.8.0c3's worse inter-block-fetch
computation time has less impact, and in a case where some servers are
more distant, slower, connected to you by narrower pipes,
malfunctioning, or more heavily loaded than other servers, then
1.8.0c3's better server-selection should be a huge win over 1.7.1.
(You can see evidence of this in
http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1170#comment:99 , where
1.8.0c3 consistently downloads a file from Test Grid at 275 KBps and
1.7.1 downloads it at 71 KBps.)

Still, I'm hoping that for v1.9, in addition to all of the other
awesome new features thanks to the Google Summer of Code projects, we
also get download pipelining (#1110 or #1187) so that v1.9 can be
significantly faster than v1.7 in every case. I'm marking #1110 with
the tag "unfinished-business"

There is one known bug in v1.8.0c3—download would bogusly abort in an
edge case that would probably arise very rarely in practice and that
we are aware of only because of a unit test. Brian and I have solved
it, but we haven't finished writing a unit test narrowly defined to
just test that case #1191.

Oh, also this weekend I added a few tests and updates to pycryptopp
and several volunteers updated their buildslaves to upload binary
.egg's of pycryptopp automatically. Hopefully this will further reduce
the installation problems of using pycryptopp. See the pycryptopp trac
instance Timeline for details.

So the overall picture is that v1.8.0 will be coming soon and it looks
like a very solid release. It has significant improvements, excellent
unit tests and code review, and it has passed a significant amount of
end-user testing during the cycle of release candidates. There are no
reports of unexplained or irreproducible bugs.

The next step is for us to finish up these issues (see the roadmap)
and release something named "v1.8.0c4". That next step will happen
either this week or next weekend. I've been experimenting with
"polyphasic sleep", which means either I'll have lots of extra time
this week to release v1.8.0c4 while the world sleeps, or else that
I'll totally crash and be zombified until I recover and won't hack on
Tahoe-LAFS again until next weekend. :-)

Needless to say, further end user testing and performance-measurement
on Tahoe-LAFS v1.8.0c3 would be very welcome. I doubt that you could
trigger the #1191 edge case, and we aren't planning on making any
other functional or performance changes for v1.8.0 final, so any
testing that you do now will apply directly to the v1.8.0 release.

Thank you very much for your help everyone!

Regards,

Zooko

[1] http://tahoe-lafs.org/pipermail/tahoe-dev/2010-September/005162.html

http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1110# pipeline download
blocks for better performance
http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1170# new-downloader
performs badly when downloading a lot of data from a file
http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1187# mitigate the
performance bottleneck of slow servers in download
http://tahoe-lafs.org/trac/tahoe-lafs/ticket/1191# unit test failure:
failed to download file with 2 shares on one server and one share on
another