Opened at 2013-08-08T13:03:42Z
Last modified at 2013-08-26T20:46:16Z
#2054 new enhancement
Track unit test duration for possible regressions
| Reported by: | markberger | Owned by: | |
|---|---|---|---|
| Priority: | normal | Milestone: | undecided |
| Component: | dev-infrastructure | Version: | 1.10.0 |
| Keywords: | Cc: | ||
| Launchpad Bug: |
Description
Currently there is no way to track regressions in tahoe over time. Ideally, real world performance testing would occur (see tickets #1406 and #1530) but we might be able to gain 80% of what we want by tracking the duration of each unit test on a dedicated machine. This would also draw attention to regressions in the test suite that were missed in code review (see ticket #2048).
The overall process would be something like this:
- Set buildbot to track master and branches with significant changes
- A developer pushes a commit to one of those branches
- Buildbot runs the test suite on a slave (possibly multiple times to get good stats) and pushes those stats to a database.
- If any significant performance regressions occurred, an email is sent to tahoe-dev or tahoe-bot complains in IRC. We could also have a bot that writes a comment on Github.
There should also be sanity checks for longer periods of time, such as a month.
To close this ticket:
- Set up a free database instance on a site like Iris Couch or MongoHQ.
- Write a script that has buildbot push stats to the database.
- Create some sort of reoccurring daemon that will notify the dev team when a regression occurs.
Some other things we might want to consider:
- Buildbot already tracks the duration of each unit test so maybe we should dump all of this historical data into the database. I'm not sure how useful it would be but we might want to back it up anyway.
- Have a web page that displays all of this information in graphs
- Since performance depends on each machine, maybe we want to create a script that iterates through the commit history and runs the unit tests on a dedicated machine.
Change History (2)
comment:1 Changed at 2013-08-19T17:18:09Z by markberger
comment:2 Changed at 2013-08-26T20:46:16Z by markberger
There is a pull request for this here: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2054
Note that this only pushes the results to https://tahoelafs.iriscouch.com and that there is nothing to analysis the data yet.
Eventually the unit test data will be uploaded here: https://tahoelafs.iriscouch.com