[tahoe-dev] new visualizer

[Brian Warner]

Now that 1.9 is out the door, I'm finally landing a summer's worth of
backlogged code. The first to arrive is a newly-rewritten Download
Status Timeline Visualizer (or "viz" for short). 1.9.0 included an early
version of this: today's patch replaces the internals to use d3.js
instead of protovis.js, which makes scrolling a lot faster. Zoom should
work properly now. There are still some improvements to be made, but
it's finally usable.

"Usable for what?", you might say. Well, it's most useful for studying
the progress and significant events in an immutable-file download. It
might help you determine why a download is running slower than you
expected. Or it might help developers find algorithmic performance
problems in the Downloader logic. You can reach the timeline by going to
the front Welcome page, following the "Recent Uploads and Downloads"
link, then choosing any "download" entry. The per-download page will
contain a big HTML table of event data, for which
docs/frontends/download-status.rst is a decent guide. There is a
"timeline" link on that table-of-numbers page which will get you to the
visualization.

The timeline shows events for a single webapi GET operation. The big
read bar shows the read() call (which requests an arbitrary span of
bytes from the file). The next section below that shows segment
requests: each read() turns into some number of fixed-size
segment-fetches. The section after that shows the individual block reads
from the various storage servers: each server gets a different color,
and you'll notice that many requests are sent to the same server in
parallel (in addition to the large chunk of block data, the downloader
also needs to gather a lot of 32-byte hashes, to perform the integrity
checks).

The "DYHB" section lives above the red read() bar: that's an
abbreviation for "Do You Have Block?" queries, which is where client
asks the appropriate storage servers if they are holding a share for
this file or not. Multiple queries are sent in parallel, and the block
fetches start as soon as shares are found.

JS hackers: please take a look at the code and see if you can improve
things: I'm sure there's a lot of work to be done. I'm going to slowly
try to add support for the other main filesystem operations (immutable
upload, mutable serverupdate, mutable publish, mutable retrieve), and
maybe eventually check/verify/repair.

cheers,
 -Brian