periodic automated test of allmydata.com prodnet grid

[zandr]

It would be valuable for operational monitoring if there were a make target that ran an upload/download test using the webapi.

This could then be run from a periodic builder and notify ops on failure, providing a real "does it work" system monitor.

[zooko]

We currently have such a buildbot-launched system test, but it currently tests the tahoe test grid, not the allmydata.com production grid.

[zooko]

I was wrong in the previous comment -- our automated test doesn't use the tahoe test grid but rather constructs a local transient grid all on the same machine, tests it, then destroys it.

We want an automated, regular test of a real live external grid. We can have one instance of such a test pointing at The Tahoe Test Grid, and another pointing at the ​http://allmydata.com production grid which holds backed up files on behalf of allmydata.com customers. Ideally this test would be sufficiently packaged that other people could use it to test their own grids.

[zooko]

Milestone 1.0.1 deleted

[warner]

Zandr is going to take responsibility for getting a few 1U machines running in the office closet. Once we have these, we'll have a host from which to run the automated prodnet test.

Zandr: please reassign to zooko once the hardware is ready.

[zandr]

Heatsinks to reactivate the old blockserver2 and blockserver4 machines have been ordered. I expect to see them Tues, so I should have hosts available on Weds.

2 machines, P4 3.0GHz, 2GB RAM, 4x250GB drives

[warner]

The "make check-grid" target now performs this sort of test, and we have buildslaves to run it against testgrid and prodnet on each trunk checkin. Another instance runs it against prodnet on each prod-branch checkin.

The only remaining bit is to rig up something to run it periodically as well (independent of changes to the source tree). I don't want to put this in the Tahoe buildbot, since it is testing the prodnet grid, not the source code. But rigging up another buildmaster might be appropriate, at least that would give us historical results and email on failure.

[zandr]

This doesn't seem inappropriate for the 'release' buildmaster, but I'm open to the idea of a 'monitoring' buildmaster as well. (or some other tool, maybe not buildbot?)

[warner]

For reference: the 'make check-grid' step (described as "check-grid prodnet" and "check-grid testgrid") is done by the 'gutsy' builder in all cases, and takes place after the unit tests and before the make-tarballs step.

[zooko]

Moving this out of the 1.3.1 Milestone.

[zooko]

I believe that we did have automated periodic monitoring of the allmydata.com prod grid, once upon a time, but that grid is now gone, so I'm closing this ticket. (Others may be interested in using the code, e.g. make check-speed and make check-grid.)