#623 closed enhancement (somebody else's problem)

coverage: compute+display differential coverage

Reported by: warner Owned by: zooko
Priority: major Milestone: eventually
Component: dev-infrastructure Version: 1.2.0
Keywords: coverage coveralls Cc:
Launchpad Bug:

Description (last modified by daira)

It would be great if the buildbot produced, for each new build, a description of lines of code that gained or lost test coverage.

I just made a change to the Makefile to include the raw figleaf pickle in the directory that gets uploaded to the webserver, so from here on out we should have machine-parsable historical data. (for the builds until this point, we only have rendered HTML for each build).

I suppose the right approach is to have a 'render-figleaf-delta' target, which starts by trying to download the pickle file from the previous build, then have a script to construct the deltas, then render those deltas into more HTML, then include the delta HTML (as well as a machine-readable delta pickle) in the next batch of uploaded data.

Change History (17)

comment:1 Changed at 2009-11-13T03:21:53Z by zooko

If you like this ticket, you may also like #810 (where did figleaf_htmlizer.py come from?), #109 (current upstream figleaf doesn't do the right thing), and #387 (figleaf confused by python2.5).

comment:2 Changed at 2009-11-24T07:02:50Z by warner

FYI, I had figleaf deltas working for a little while, but the results weren't very helpful: they were too easily tricked by insertion/removal of lines. If I were going to reimplement this code again now, I would try to be more clever. Maybe compare the contents of lines (if you think a line has lost coverage, compare it against all lines which have gained coverage, and allow identical lines to cancel each other out). Or maybe find a clever 'diff' algorithm that spots insertions.

Also, the report should only show files which have had their coverage change. We don't need yet another giant report causing users to have to scroll a lot to find the interesting parts.

comment:3 Changed at 2009-11-24T15:12:10Z by zooko

Maybe use the darcs patch instead of diffing the before- and after- versions of the file.

comment:4 Changed at 2009-11-24T15:41:09Z by zooko

Oh, in fact maybe what you really want to know is several different things:

  • lines removed that were covered
  • lines removed that were uncovered :-)
  • lines added and covered
  • lines added and uncovered :-(
  • lines not touched but went from covered to uncovered :-(
  • lines not touched but went from uncovered to covered :-)

comment:5 Changed at 2010-01-27T23:13:58Z by zooko

  • Milestone changed from undecided to 1.7.0

comment:6 Changed at 2010-01-27T23:30:13Z by zooko

  • Owner changed from somebody to zooko
  • Status changed from new to assigned

comment:7 Changed at 2010-02-27T18:15:25Z by zooko

  • Summary changed from figleaf: compute+display differential coverage to coverage: compute+display differential coverage

comment:8 Changed at 2010-03-17T00:42:01Z by davidsarah

  • Keywords coverage added

comment:9 Changed at 2010-06-16T02:58:22Z by zooko

  • Milestone changed from 1.7.0 to eventually

comment:10 Changed at 2014-09-01T14:29:22Z by Daira Hopwood <daira@…>

In 61371a31df3c89afe1f7b51ac6fd7e94f3fc49f4/trunk:

Support coverage reporting via coverage.io. refs #623

Signed-off-by: Daira Hopwood <daira@…>

comment:11 Changed at 2014-09-09T18:31:00Z by daira

  • Description modified (diff)
  • Keywords coveralls added
  • Resolution set to fixed
  • Status changed from assigned to closed

Actually coveralls.io: https://coveralls.io/r/tahoe-lafs/tahoe-lafs

This isn't using buildbot but I think it fulfills the spirit of this ticket.

comment:12 Changed at 2014-09-09T18:35:09Z by daira

Hmm, actually coveralls.io reports display the number of lines by which coverage changed for each file, but they don't display a diff of the coverage change. Perhaps we should file an enhancement request for that.

comment:13 Changed at 2014-09-09T19:38:11Z by daira

Filed https://github.com/lemurheavy/coveralls-public/issues/359.

Also see https://github.com/lemurheavy/coveralls-public/issues/31 for display of branch coverage. (We collect the per-line branch coverage information but it's not displayed on coveralls.)

comment:14 Changed at 2014-09-09T19:41:00Z by daira

  • Resolution fixed deleted
  • Status changed from closed to reopened

Reopening until coveralls supports differential coverage reports (or we find another way of doing this).

comment:15 Changed at 2014-09-10T08:14:13Z by daira

http://nedbatchelder.com/code/coverage/branch.html says:

The XML report produced by coverage xml also includes branch information, including separate statement and branch coverage percentages.

Here is an example of how the branch data is represented in the XML format; it looks quite comprehensive.

So, the lack of branch coverage in the coveralls reports is purely a problem with the rendering done by coveralls.io.

Version 0, edited at 2014-09-10T08:14:13Z by daira (next)

comment:16 Changed at 2014-09-10T08:15:14Z by daira

  • Resolution set to somebody else's problem
  • Status changed from reopened to closed

comment:17 Changed at 2014-09-10T09:27:42Z by daira

Aside -- this is how detailed code coverage *should* be: Haskell example

Note: See TracTickets for help on using tickets.