[tahoe-lafs-trac-stream] [Tahoe-LAFS] #3138: Run the test suite concurrency on CircleCI

[Tahoe-LAFS]

#3138: Run the test suite concurrency on CircleCI
------------------------------------+-----------------------
     Reporter:  exarkun             |      Owner:  exarkun
         Type:  defect              |     Status:  new
     Priority:  normal              |  Milestone:  undecided
    Component:  dev-infrastructure  |    Version:  n/a
   Resolution:                      |   Keywords:
Launchpad Bug:                      |
------------------------------------+-----------------------
Description changed by exarkun:

Old description:

> CircleCI provides runners with two vCPUs.  If a single CircleCI job can
> split its work across two Python processes we can take better advantage
> of the runner configuration and hopefully get a much faster CI run.
>
> trial supports spreading test execution over multiple local processes
> with `--jobs`.  We can already pass extra arguments to trial through tox
> thanks to #2894.
>
> There are some problems with coverage collection's interactions with
> `--jobs`.  There is a Twisted ticket to deal with this,
> https://twistedmatrix.com/trac/ticket/9663, but there is also
> https://pypi.org/project/coverage_enable_subprocess/ which basically
> solves the problem externally.
>
> So, add coverage_enable_subprocess and `--jobs 2` to our CircleCI
> configuration and get a ~50% reduction in CircleCI runtime.
>

> See #2845 and #2894 for past ideas in this space.  See  for a Twisted
> ticket meant to support this effort.

New description:

 CircleCI provides runners with two vCPUs.  If a single CircleCI job can
 split its work across two Python processes we can take better advantage of
 the runner configuration and hopefully get a much faster CI run.

 trial supports spreading test execution over multiple local processes with
 `--jobs`.  We can already pass extra arguments to trial through tox thanks
 to #2894.

 There are some problems with coverage collection's interactions with
 `--jobs`.  There is a Twisted ticket to deal with this,
 https://twistedmatrix.com/trac/ticket/9663, but there is also
 https://pypi.org/project/coverage_enable_subprocess/ which basically
 solves the problem externally.

 So, add coverage_enable_subprocess and `--jobs 2` to our CircleCI
 configuration and get a ~50% reduction in CircleCI runtime.

--

--
Ticket URL: <https://tahoe-lafs.org/trac/tahoe-lafs/ticket/3138#comment:1>
Tahoe-LAFS <https://Tahoe-LAFS.org>
secure decentralized storage