[tahoe-dev] Benchmarking script

[Kyle Markley]

On Mon, 16 Aug 2010 16:46:49 -0600, "Zooko O'Whielacronx"
<zooko at zooko.com>
wrote:
> Kyle:
> 
> Could we set up a time to get together on IRC and chat about automated
> performance measurements. The last-minute bug in the v1.8 downloader
> has convinced me that we need to get that stuff fully hooked into
> buildbot so that we always find out about performance regressions
> within a few minutes after they are committed to revision control.

Sure; let's set up a time off-list.


> Your script is probably a good start, but I'm not sure how much it
> would take to make it a fully autonomous, buildbot-managed service.

A lot, I would think, if we want the buildbots to use real (remote) grids.
I didn't do anything at all regarding security, denial of service, or
connecting through NAT, and we'd also have the thorn of all the nodes
wanting to run benchmarks at nearly the same time.  (The current script
would serialize them, which might be the right behavior.)

It may be a lot easier to start by benchmarking purely local operations. 
We'd have CPU/disk data but not network data, but that's still better than
nothing.

... or maybe we'd want to use the test grid instead of creating a
temporary grid on the fly?  It may be hard to reproduce and trust our
measurements on the test grid, but on the other hand it would be very easy
to use!

-- 
Kyle Markley