[tahoe-lafs-trac-stream] [Tahoe-LAFS] #4126: CI test_system fails inconsistently (was: Tests time out on CircleCI, subsequent 'unclean reactor' errors)

[Tahoe-LAFS]

#4126: CI test_system fails inconsistently
------------------------------------+---------------------------
     Reporter:  hacklschorsch       |      Owner:  hacklschorsch
         Type:  defect              |     Status:  assigned
     Priority:  normal              |  Milestone:  undecided
    Component:  dev-infrastructure  |    Version:  n/a
   Resolution:                      |   Keywords:  ci
Launchpad Bug:                      |
------------------------------------+---------------------------
Changes (by btlogy):

 * keywords:   => ci
 * component:  unknown => dev-infrastructure


Old description:

> In https://github.com/tahoe-lafs/tahoe-
> lafs/pull/1381#issuecomment-2476885548 meejah writes:
>
> > The unclean-reactor errors may be simply a downstream symptom of the
> real errors that also happen in that run (e.g. several tests time out).
>
> My own tests suggest that indeed, raising the SystemTests timeout make
> [https://github.com/tahoe-lafs/tahoe-
> lafs/pull/1381#issuecomment-2444698978 a couple of flaky tests] much more
> stable:
>
> || Failure count || Test name ||
> ||     1 || allmydata.test.test_system.HTTPSystemTest.test_mutable_mdmf
> ||
> ||     3 || allmydata.test.test_system.HTTPSystemTest.test_mutable_sdmf
> ||
> ||    30 ||
> allmydata.test.test_system.HTTPSystemTest.test_upload_and_download_convergent
> ||
> ||    11 ||
> allmydata.test.test_system.HTTPSystemTest.test_upload_and_download_random_key
> ||
>
> This ticket is similar but not equal to ticket:4085, ticket:4022,
> ticket:2994 .

New description:

 CI reactors under `test.test_system` on CircleCI fail inconsistently ONLY
 in the Tahoe-lafs Circle CI org.
 Cannot reproduce locally on Nixos nor on GitHub CI (inside similar docker
 images).

 Possible root cause discussed in https://github.com/tahoe-lafs/tahoe-
 lafs/pull/1381#issuecomment-2476885548 meejah writes:

 > The unclean-reactor errors may be simply a downstream symptom of the
 real errors that also happen in that run (e.g. several tests time out).

 My own tests suggest that indeed, raising the SystemTests timeout make
 [https://github.com/tahoe-lafs/tahoe-
 lafs/pull/1381#issuecomment-2444698978 a couple of flaky tests] much more
 stable:

 || Failure count || Test name ||
 ||     1 || allmydata.test.test_system.HTTPSystemTest.test_mutable_mdmf ||
 ||     3 || allmydata.test.test_system.HTTPSystemTest.test_mutable_sdmf ||
 ||    30 ||
 allmydata.test.test_system.HTTPSystemTest.test_upload_and_download_convergent
 ||
 ||    11 ||
 allmydata.test.test_system.HTTPSystemTest.test_upload_and_download_random_key
 ||

 This ticket is similar but not equal to ticket:4085, ticket:4022,
 ticket:2994 .

--

Comment:

 If I'm not mistaken, those reactor

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/4126#comment:6>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage