Changes between Version 2 and Version 3 of HowToWriteTests

Timestamp:

2012-03-30T23:06:25Z (12 years ago)

Author:

zooko

Comment:

instructions on the Why of code coverage

HowToWriteTests

@@ -14 +14 @@
 
 {{{
-rom twisted.trial import unittest
+from twisted.trial import unittest
 
 class T(unittest.TestCase):
@@ -20 +20 @@
         pass
 }}}
+
+Now run it!
+
+= code coverage =
 
 Now install Ned Batchelder's "[http://nedbatchelder.com/code/coverage/ coverage"] tool and run your with code coverage, like this:
@@ -40 +44 @@
 
 That will product a directory named {{{htmlcov}}}. View its contents with a web browser.
+
+= using code coverage results =
+
+This is important: we do not treat code coverage numbers as a litmus test (like "aim to have 90% of lines covered"). We hardly even treat it as a scalar measurement of goodness -- 91% code coverage is not necessarily better than 90% code coverage. Maybe the alternative would have been to remove some (covered) lines of code that were not necessary, which would have resulted in a worse "code coverage" metric but a better codebase. Finally, note that even if you have 100% branch-level coverage of a codebase, that doesn't mean that your tests are exercising all possible ways that the codebase could be run! There could be data-dependent bugs, such as a divide-by-zero error, or a path which sets one variable to a setting which is inconsistent with a different variable. These sorts of bugs might not be getting exercised by the test code even though every line and every branch of the code is getting tested.
+
+So what do we use it for? It is a lens through which to view your code and your test code. You should look at the code coverage results and think about what it says about your tests. Think about "what could go wrong" in this function -- where bugs could be in this function or a future version of it -- and whether the current tests would catch those bugs. Both authors of patches and reviewers of patches should look at the code coverage results, and see if they indicate important holes in the tests.
+
+Likewise, even if the code coverage shows maximal coverage, you should *still* think "Are there any kinds of bugs that could exist in this or a future version of this that *wouldn't* be caught by these tests?". 
+
+= further reading =
+
+http://twistedmatrix.com/documents/current/core/howto/testing.html