#1532 closed defect (wontfix)

test suite: measure time taken by a representative test, and set other timeouts based on that

Reported by: davidsarah Owned by: somebody
Priority: major Milestone: undecided
Component: code Version: 1.9.0a1
Keywords: test timeout Cc:
Launchpad Bug:

Description

Brian suggested this idea on IRC. It would alleviate to some extent the problem of having to set ridiculously long timeouts to accomodate slow buildslaves, and then not being able to get results quickly from fast buildslaves (or local testing) when there is a bug that causes a hang.

Change History (3)

comment:1 follow-up: Changed at 2011-09-14T04:13:19Z by warner

BTW, I was half-kidding.. I'd be concerned about the uncertainty that would introduce. OTOH, maybe we just disable timeouts altogether on the super-slow buildslaves. I think it's important to have timeouts on the normal buildslaves, because an asynchronous system as complex as Tahoe is always going to run into lost-progress bugs from time to time, and it'd be a shame for us to have to manually restart a buildslave to deal with them.

comment:2 in reply to: ↑ 1 Changed at 2011-09-14T14:55:18Z by davidsarah

Replying to warner:

BTW, I was half-kidding.. I'd be concerned about the uncertainty that would introduce. OTOH, maybe we just disable timeouts altogether on the super-slow buildslaves. I think it's important to have timeouts on the normal buildslaves, because an asynchronous system as complex as Tahoe is always going to run into lost-progress bugs from time to time, and it'd be a shame for us to have to manually restart a buildslave to deal with them.

If there is a lost-progress bug, it's just as likely to affect the slow buildslaves as the fast ones. So I think that in practice, disabling timeouts for the slow buildslaves would just cause them to become stuck and need manual intervention more often.

comment:3 Changed at 2020-01-16T20:51:30Z by exarkun

  • Resolution set to wontfix
  • Status changed from new to closed

I don't think there is any such thing as a "representative test". To the greatest extent possible, the tests should be exercising *different* code. There is no value in re-executing the same code paths multiple times. If a code path worked the first time, it will work a second time. What's valuable is testing *different* code paths.

Also, the outlier-slow CI jobs are gone now and test suite timeouts generally haven't been an issue lately.

Note: See TracTickets for help on using tickets.