test suite: measure time taken by a representative test, and set other timeouts based on that

[davidsarah]

Brian suggested this idea on IRC. It would alleviate to some extent the problem of having to set ridiculously long timeouts to accomodate slow buildslaves, and then not being able to get results quickly from fast buildslaves (or local testing) when there is a bug that causes a hang.

[warner]

BTW, I was half-kidding.. I'd be concerned about the uncertainty that would introduce. OTOH, maybe we just disable timeouts altogether on the super-slow buildslaves. I think it's important to have timeouts on the normal buildslaves, because an asynchronous system as complex as Tahoe is always going to run into lost-progress bugs from time to time, and it'd be a shame for us to have to manually restart a buildslave to deal with them.

[davidsarah]

Replying to warner:

BTW, I was half-kidding.. I'd be concerned about the uncertainty that would introduce. OTOH, maybe we just disable timeouts altogether on the super-slow buildslaves. I think it's important to have timeouts on the normal buildslaves, because an asynchronous system as complex as Tahoe is always going to run into lost-progress bugs from time to time, and it'd be a shame for us to have to manually restart a buildslave to deal with them.

If there is a lost-progress bug, it's just as likely to affect the slow buildslaves as the fast ones. So I think that in practice, disabling timeouts for the slow buildslaves would just cause them to become stuck and need manual intervention more often.

[exarkun]

I don't think there is any such thing as a "representative test". To the greatest extent possible, the tests should be exercising *different* code. There is no value in re-executing the same code paths multiple times. If a code path worked the first time, it will work a second time. What's valuable is testing *different* code paths.

Also, the outlier-slow CI jobs are gone now and test suite timeouts generally haven't been an issue lately.