[tahoe-dev] Tahoe-LAFS v1.6: only one ticket left open

[Zooko Wilcox-O'Hearn]

Folks:

We just committed the fix for #833 -- thanks to David-Sarah for  
writing the patch and Kevan for reviewing it.  #833 makes it so that  
you have the "deep-immutable" guarantee that you would expect from an  
immutable directory: if you start in an immutable directory then  
*everything* you can reach from that starting point is also  
immutable, and you can rely on the fact that anyone else who starts  
in the same directory will also see the same things.

#833 also adds some very nice forward-compatibility features, so that  
when future versions of Tahoe-LAFS start uploading new kinds of  
capabilities, people using Tahoe-LAFS v1.6 will get a reasonable  
indicator showing that there is something here that is immutable, but  
is from the future so you can't actually see it, instead of getting a  
traceback or some sort of behavior that violates the deep- 
immutability property.  (This is important because Tahoe-LAFS v1.6 is  
going into Ubuntu 10.04, and hopefully also into the next release of  
Fedora, so there will be many Tahoe-LAFS v1.6 users for a long time  
to come.) See the "Memory Lane" matrix of features vs. Tahoe-LAFS  
versions on The TestGrid Page [1] for an example of what a cap from  
the future looks like to Tahoe-LAFS v1.6.

That leaves only one ticket open for v1.6, #928.  See The Roadmap [2]  
for the 44 closed tickets.

#928 is a good robustness, performance, and load-balancing  
improvement for download.  In current trunk, whenever you download a  
file your Tahoe-LAFS storage client starts by querying every storage  
server on the grid to find out what shares each server has.  Then it  
waits for every query to be answered (or for the storage server to  
drop its TCP connection instead of answering the query).  Then it  
chooses what is basically a random K shares from the set of all  
shares that it learned about and downloads those.  After #928, it  
will not wait to hear the answer to *every* query, but will wait only  
until it has learned about K shares.  As soon as it learns about K  
shares it will proceed to download the file using those.

This has three beneficial effects: first, if there is a server on the  
grid which is in a hung state such that it keeps its TCP connections  
open but doesn't answer queries, then with current trunk all  
downloads from that grid will stall indefinitely.  With #928, that  
hung server will have no deleterious effect on downloads at all  
(provided that K shares can be found on other servers).  Second, the  
servers which answer the query fastest are the ones that the  
downloader uses, which hopefully means that the servers that have a  
faster connection to the downloader are the ones that get used, so:  
faster downloads! This effect should be more pronounced on grids with  
heterogeneous and geographically dispersed servers than on grids with  
a bunch of homogeneous servers on the same LAN.  Third, servers that  
are busier will tend to answer the queries later, so this patch makes  
downloaders automatically load-balance to download shares from  
servers that are less busy.

It is a nice simple patch, written by me, and reviewed and approved  
by Brian, but it isn't in trunk yet because it doesn't yet have  
thorough unit tests.  (David-Sarah has posted the beginnings of unit  
tests and they or I will finish them soon.)  If you know how to use  
darcs, you can get the most recent darcspatch from #928, apply it,  
try it out, and tell us whether downloads go any faster for you with  
#928 than without.

As soon as we finish thorough unit tests for #928 I'll commit it to  
trunk. In the meantime any reports from the field of success or  
failures with current trunk or with current trunk+#928 would be good  
to have.  Tahoe-LAFS v1.6 is going to be an extremely reliable,  
stable new release with major feature and performance improvements.  
Exciting!

After #928, the only remaining tasks before release are to write up  
relnotes.txt, build binary packages for as many platforms as we can  
(help!!), and send the release announcement to mailing lists and news  
sites that might be interested.

Thank you all very much for your support of this project.

Regards,

Zooko

[1] http://allmydata.org/trac/tahoe/wiki/TestGrid
[2] http://allmydata.org/trac/tahoe/roadmap

http://allmydata.org/trac/tahoe/ticket/833# reject mutable children  
when *reading* an immutable dirnode
http://allmydata.org/trac/tahoe/ticket/928# start downloading as soon  
as you know where to get K shares