[tahoe-lafs-trac-stream] [Tahoe-LAFS] #3024: Speed up CircleCI builds by caching more of the environment setup

[Tahoe-LAFS]

#3024: Speed up CircleCI builds by caching more of the environment setup
-------------------------+-----------------------
     Reporter:  exarkun  |      Owner:
         Type:  defect   |     Status:  new
     Priority:  normal   |  Milestone:  undecided
    Component:  unknown  |    Version:  1.12.1
   Resolution:           |   Keywords:
Launchpad Bug:           |
-------------------------+-----------------------
Description changed by exarkun:

Old description:

> Most CircleCI builds proceed like this:
>
> 1. Boot stock OS Docker image
> 2. Update OS package manager
> 3. Get the code
> 4. Install OS dependencies
> 5. Restore the pip http cache
> 6. Restore the pip wheel cache
> 7. Create the tox virtualenv and install Python dependencies in it
> 8. Save the pip wheel cache
> 9. Save the pip http cache
> 10. Run the tests
> 11. Upload the test logs and coverage data
>
> It is pretty unavoidable that we will have to boot some kind of Docker
> image.
> Except for (7), steps (2) through (9) are pretty much pure redundant
> overhead.  There's almost never any change from run to run in what they
> do.  Possibly step (7) isn't worth an exception from this list, either.
>
> From a recent master job, steps (10) and (11) compared to the whole job
> run were (non-test % in parens):
>
> * ubuntu 18.04:    8m44s vs 7m18s (16%)
> * debian 9:        8m37s vs 7m37s (11%)
> * centos 7:        9m43s vs 8m36s (11%)
> * c-locale:       10m37s vs 7m32s (29%)
> * deprecations:   10m25s vs 8m44s (16%)
> * fedora 29:       9m30s vs 7m22s (22%)
> * slackware 14.2:  9m23s vs 7m54s (15%)
> * another-locale:  8m37s vs 7m35s (12%)
> * fedora 28:       9m26s vs 7m31s (20%)
> * debian 8:        8m13s vs 7m02s (14%)
> * ubuntu 16.04:    8m36s vs 7m16s (15%)
>
> Two other jobs have a different structure but we can make a similar
> comparison:
>
> * lint:            1m54s vs 1m48s (5%)
> * integration:     4m46s vs 3m35s (24%)
>
> It seems reasonable to suggest that if we can factor out the redundant
> setup we should see at least a 10% speedup in overall CI runtime -
> perhaps as much as 20% or 30% (when CircleCI is under load, I guess?)

New description:

 Most CircleCI builds proceed like this:

 1. Boot stock OS Docker image
 2. Update OS package manager
 3. Get the code
 4. Install OS dependencies
 5. Restore the pip http cache
 6. Restore the pip wheel cache
 7. Create the tox virtualenv and install Python dependencies in it
 8. Save the pip wheel cache
 9. Save the pip http cache
 10. Run the tests
 11. Upload the test logs and coverage data

 It is pretty unavoidable that we will have to boot some kind of Docker
 image.
 Except for (7), steps (2) through (9) are pretty much pure redundant
 overhead.  There's almost never any change from run to run in what they
 do.  Possibly step (7) isn't worth an exception from this list, either.

 From a recent master job, steps (10) and (11) compared to the whole job
 run were (non-test % in parens):

 {{{
 * ubuntu 18.04:    8m44s vs 7m18s (16%)
 * debian 9:        8m37s vs 7m37s (11%)
 * centos 7:        9m43s vs 8m36s (11%)
 * c-locale:       10m37s vs 7m32s (29%)
 * deprecations:   10m25s vs 8m44s (16%)
 * fedora 29:       9m30s vs 7m22s (22%)
 * slackware 14.2:  9m23s vs 7m54s (15%)
 * another-locale:  8m37s vs 7m35s (12%)
 * fedora 28:       9m26s vs 7m31s (20%)
 * debian 8:        8m13s vs 7m02s (14%)
 * ubuntu 16.04:    8m36s vs 7m16s (15%)
 }}}

 Two other jobs have a different structure but we can make a similar
 comparison:

 {{{
 * lint:            1m54s vs 1m48s (5%)
 * integration:     4m46s vs 3m35s (24%)
 }}}

 It seems reasonable to suggest that if we can factor out the redundant
 setup we should see at least a 10% speedup in overall CI runtime - perhaps
 as much as 20% or 30% (when CircleCI is under load, I guess?)

--

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3024#comment:1>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage