Unit test suite inconsistently failing on CircleCI

[hacklschorsch]

1. For at least 3 months (likely more, but can no longer see older logs) we can see test_verify_one_bad_encprivkey spuriously failing in the CircleCI logs (except for master which was broken #4098)
2. More recently, test_system.HTTPSystemTest is failing more often:

CI reactors under test.test_system on CircleCI fail inconsistently in the Tahoe-lafs AND LeastAuthority? orgs (not the same plan). And this cannot be reproduced locally on Nixos nor on GitHub? CI (inside similar docker images).

Possible root cause discussed in ​https://github.com/tahoe-lafs/tahoe-lafs/pull/1381#issuecomment-2476885548 meejah writes:

The unclean-reactor errors may be simply a downstream symptom of the real errors that also happen in that run (e.g. several tests time out).

My own tests suggest that indeed, raising the SystemTests? timeout make ​a couple of flaky tests much more stable:

Failure count	Test name
1	allmydata.test.test_system.HTTPSystemTest.test_mutable_mdmf
3	allmydata.test.test_system.HTTPSystemTest.test_mutable_sdmf
30	allmydata.test.test_system.HTTPSystemTest.test_upload_and_download_convergent
11	allmydata.test.test_system.HTTPSystemTest.test_upload_and_download_random_key

This ticket is similar but not equal to ticket:4085, ticket:4022, ticket:2994 .

NOTE: there is an ongoing collaborative effort to get to the bottom of this issue using this tmp doc: ​https://cryptpad.fr/code/#/2/code/view/ApS8GZH4OfKbR71RdkRa1LLClaJk88emHeW0yvwhHkk/

[hacklschorsch]

A better fix to the dirty reactors might be to clean them up - and indeed, SystemTests does some reactor-cleanup dance in tearDown(), which ​*should* also be called if a test fails - but it seems that does not always happen or always help.

This is called even if the test method raised an exception [...] This method will only be called if the setUp() succeeds

Also see the discussion in ​https://stackoverflow.com/questions/39883058/teardown-not-called-after-timeout-in-twisted-trial

[hacklschorsch]

Raising timeout in ​https://github.com/tahoe-lafs/tahoe-lafs/pull/1387

[btlogy]

If I'm not mistaken, those reactor

[btlogy]

I'm also observing some similar inconsistent failure on GHA: ​https://github.com/tahoe-lafs/tahoe-lafs/pull/1404#issuecomment-2524644247

• ubuntu-20.04, py-3.11: FAILED integration/test_web.py::test_upload_download

ValueError: Expected a 2xx code, got 500

• macos-14, py-3.11: ERROR integration/test_tor.py::test_onion_service_storage

twisted.internet.error.ProcessTerminated: A process has ended with a probable error condition: process ended with exit code 255

[btlogy]

Also be happening in Windows:

​https://app.circleci.com/pipelines/github/LeastAuthority/tahoe-lafs/1216/workflows/27697f8b-e55e-4a4d-953a-09c2dfbce129/jobs/14801/tests#failed-test-0

allmydata.test.test_system.HTTPSystemTest: test_mutable_sdmf

allmydata.mutable.common.NotEnoughServersError: ("Publish ran out of good servers, last failure was: [Failure instance: Traceback: <class 'twisted.internet.error.ConnectingCancelledError'>

[hacklschorsch]

Some useful statistics on which tests fail the most - and a not so useful since not working for us? flaky-test detection - can be found here: ​https://app.circleci.com/insights/github/tahoe-lafs/tahoe-lafs/workflows/ci/tests

• "Flaky tests" is empty (?)
• "Most Failed Tests" contain our flaky tests (which probably are failing because of earlier problems)
• The first four or so pages (a ten tests) of "Slowest Tests" include basically all the SystemTest and !HTTPSystemTest suites that fail randomly and too often and probably should be integration tests instead (?)