#3391 closed enhancement (fixed)

Change codecov default settings

Reported by: sajith Owned by: sajith
Priority: normal Milestone: undecided
Component: dev-infrastructure Version: n/a
Keywords: Cc:
Launchpad Bug:

Description

Codecov.io has been a source of unhappiness lately, with their test coverage checks leaving red check marks in CI status in a rather arbitrary manner: example, example.

It is not clear why "91.47% (+-0.06%) compared to base commit" or "91.49% (+-0.03%) compared to base commit" is a bad thing. My thinking is that these are not large enough changes in coverage to warrant CI turning red, and maybe this is happening because computers are not very good with floating point numbers. We will ask codecov to drop the decimals and turn a few other knobs by adding a codecov.yml file (this is quite well documented) to the repository and see how that works for us.

Possibly related tickets: #3384, #3385.

Change History (6)

comment:1 Changed at 2020-08-22T21:56:36Z by sajith

  • Keywords review-needed added

comment:2 Changed at 2020-08-25T15:56:20Z by exarkun

Just sharing, the codecov docs at https://docs.codecov.io/docs/troubleshooting-guide say you can get the actual coverage reports its operating on from the "Builds" tab (note it's a tab on the commits page). This gives you a page with a bunch of download links, eg https://codecov.io/gh/tahoe-lafs/tahoe-lafs/commit/bce701d5bc810b71c07d2b7142a59aaac28a45e4/build, which could then presumably be locally inspected to see if codecov is wrong or the coverage reports are actually different.

comment:3 Changed at 2020-09-02T19:39:59Z by meejah

  • Keywords review-needed removed

comment:4 Changed at 2020-09-15T21:17:13Z by sajith

Those coverage reports look odd to me. In this specific instance:

I'm not sure if this happens because coverage.xml wasn't generated on CircleCI runs, or because of any issues with codecov uploader itself.

comment:5 Changed at 2020-09-15T21:31:42Z by exarkun

Maybe we should add some more logging to the CI configuration to show us what coverage does get generated by each job - before it is supposed to be uploaded to codecov?

comment:6 Changed at 2020-09-15T21:33:59Z by GitHub <noreply@…>

  • Resolution set to fixed
  • Status changed from new to closed

In 1b207d8/trunk:

Merge pull request #789 from sajith/3391.codecov-yml

Change codecov defaults

Fixes ticket:3391

Note: See TracTickets for help on using tickets.