Changes between Version 5 and Version 6 of HowToWriteTests


Ignore:
Timestamp:
2013-08-07T17:57:28Z (8 years ago)
Author:
zooko
Comment:

update rationale for code coverage, and adjust formatting

Legend:

Unmodified
Added
Removed
Modified
  • HowToWriteTests

    v5 v6  
    4747= using code coverage results =
    4848
    49 This is important: we do not treat code coverage numbers as a litmus test (like "aim to have 90% of lines covered"). We hardly even treat it as a scalar measurement of goodness -- 91% code coverage is not necessarily better than 90% code coverage. Maybe the alternative would have been to remove some (covered) lines of code that were not necessary, which would have resulted in a worse "code coverage" metric but a better codebase. Finally, note that even if you have 100% branch-level coverage of a codebase, that doesn't mean that your tests are exercising all possible ways that the codebase could be run! There could be data-dependent bugs, such as a divide-by-zero error, or a path which sets one variable to a setting which is inconsistent with a different variable. These sorts of bugs might not be getting exercised by the test code even though every line and every branch of the code is getting tested.
     49This is important: we do not treat code coverage numbers as a litmus test (like "aim to have 90% of lines covered"). We hardly even treat it as a scalar measurement of goodness — 91% code coverage is not necessarily better than 90% code coverage. Maybe the alternative would have been to remove some (covered) lines of code that were not necessary, which would have resulted in a worse “code coverage” metric but a better codebase. Finally, note that even if you have 100% branch-level coverage of a codebase, that doesn't mean that your tests are exercising all possible ways that the codebase could be run! There could be data-dependent bugs, such as a divide-by-zero error, or a path which sets one variable to a setting which is inconsistent with a different variable. These sorts of bugs might not be getting exercised by the test code even though every line and every branch of the code is getting tested.
    5050
    51 So what do we use it for? It is a lens through which to view your code and your test code. You should look at the code coverage results and think about what it says about your tests. Think about "what could go wrong" in this function -- where bugs could be in this function or a future version of it -- and whether the current tests would catch those bugs. Both authors of patches and reviewers of patches should look at the code coverage results, and see if they indicate important holes in the tests.
     51So what do we use it for? It is a lens through which to view your code and your test code. You should look at the code coverage results and think about what it says about your tests. Think about “what could go wrong” in this function — where bugs could be in this function or a future version of it — and whether the current tests would catch those bugs. Both authors of patches and reviewers of patches should look at the code coverage results, and see if they indicate important holes in the tests.
    5252
    53 Likewise, even if the code coverage shows maximal coverage, you should *still* think "Are there any kinds of bugs that could exist in this or a future version of this that *wouldn't* be caught by these tests?".
     53Code coverage displays turn out to be very handy for showing you facts about your tests and your code that you didn't know.
    5454
    5555= further reading =