#3601 new defect

CI builds new images for test jobs even when Tahoe's dependencies haven't changed

Reported by: exarkun Owned by:
Priority: normal Milestone: undecided
Component: unknown Version: n/a
Keywords: self-contained ci Cc:
Launchpad Bug:

Description

Currently on CircleCI we use pre-built Docker images populated with our dependencies in order to make build results more easily reproducible and to speed up individual test jobs.

The Docker images are built and replaced nightly:

https://github.com/tahoe-lafs/tahoe-lafs/blob/94716c078a5c9cbaf468b88206d911146ba6091e/.circleci/config.yml#L97-L129

There are some downsides to replacing the images this way:

  • Often Tahoe's dependencies have not changed. The introduction of a new image only allows for outside factors to lead to changes in behavior. This reduces reproducibility since it means the same Tahoe source tested on two different days might encounter two different environments and end up with two different results.
  • It's wasteful to build images when nothing has changed. It's wasteful of CircleCI's resources (which we currently consume for free). It's wasteful of our time since re-built images are less likely to be cached on build hosts and need to be downloaded which takes time.

For a set of "stable" CI jobs which should only fail if Tahoe introduces a regression, not if the outside world changes in a way that breaks Tahoe, the ideal would be to only rebuild the images when Tahoe's dependencies change, only allow the new image to become the canonical image if the Tahoe works on the image, and only allow Tahoe dependencies to be resolved during image creation, not during test execution.

Change History (2)

comment:1 Changed at 2021-01-27T14:07:26Z by exarkun

  • Keywords self-contained ci added

comment:2 Changed at 2023-02-13T14:27:37Z by exarkun

The CI images aren't build nightly anymore (as of ticket:3958) but it's still possible to use the rebuild-images.sh tool to introduce regressions by pulling in modified dependencies.

Note: See TracTickets for help on using tickets.