#3024 closed defect

Speed up CircleCI builds by caching more of the environment setup — at Version 1

Reported by: exarkun Owned by:
Priority: normal Milestone: undecided
Component: unknown Version: 1.12.1
Keywords: review-needed Cc:
Launchpad Bug:

Description (last modified by exarkun)

Most CircleCI builds proceed like this:

  1. Boot stock OS Docker image
  2. Update OS package manager
  3. Get the code
  4. Install OS dependencies
  5. Restore the pip http cache
  6. Restore the pip wheel cache
  7. Create the tox virtualenv and install Python dependencies in it
  8. Save the pip wheel cache
  9. Save the pip http cache
  10. Run the tests
  11. Upload the test logs and coverage data

It is pretty unavoidable that we will have to boot some kind of Docker image. Except for (7), steps (2) through (9) are pretty much pure redundant overhead. There's almost never any change from run to run in what they do. Possibly step (7) isn't worth an exception from this list, either.

From a recent master job, steps (10) and (11) compared to the whole job run were (non-test % in parens):

* ubuntu 18.04:    8m44s vs 7m18s (16%)
* debian 9:        8m37s vs 7m37s (11%)
* centos 7:        9m43s vs 8m36s (11%)
* c-locale:       10m37s vs 7m32s (29%)
* deprecations:   10m25s vs 8m44s (16%)
* fedora 29:       9m30s vs 7m22s (22%)
* slackware 14.2:  9m23s vs 7m54s (15%)
* another-locale:  8m37s vs 7m35s (12%)
* fedora 28:       9m26s vs 7m31s (20%)
* debian 8:        8m13s vs 7m02s (14%)
* ubuntu 16.04:    8m36s vs 7m16s (15%)

Two other jobs have a different structure but we can make a similar comparison:

* lint:            1m54s vs 1m48s (5%)
* integration:     4m46s vs 3m35s (24%)

It seems reasonable to suggest that if we can factor out the redundant setup we should see at least a 10% speedup in overall CI runtime - perhaps as much as 20% or 30% (when CircleCI is under load, I guess?)

Change History (1)

comment:1 Changed at 2019-04-03T15:09:45Z by exarkun

  • Description modified (diff)
Note: See TracTickets for help on using tickets.