There’s Only One Thing You Can Learn From Code Coverage
Posted by Davy Brion on October 15th, 2009
When you hear people talk about automated testing, the subject of code coverage typically comes up as well. Code coverage is a metric which indicates how much of your production code is executed while your automated tests are running. A lot of people think that high code coverage is a good thing, because it means that a high percentage of their production code is being executed by their tests. That’s gotta be a good thing, right?
Well, not necessarily. You can very easily write bad tests which execute most, or even all of your production code, while the tests themselves probably don’t cover everything that really needs to be covered. Even if you’re a disciplined developer who is very careful not to make mistakes, the odds that your code will contain bugs that your tests didn’t catch are still relatively likely. For those of you who write automated tests (either test-first or test-after): how often has it happened to you that a bug was discovered later on for some edge case that you didn’t think of when writing the production code or the test code? No matter how good you are, this has happened to you on more than one occasion. And it will happen again in the future.
Here’s the important question though: how frequently does it happen that bugs are discovered in pieces of code which have high coverage during the test-runs? It might not happen every day, it might not happen every week but it definitely happens occasionally and in some cases even frequently. What exactly does that tell you about the value of a metric such as code coverage? You might think that if this situation comes up, you don’t have proper tests in place but that’s not necessarily the case. You can have an incredibly good test fixture (or several) for a certain piece of code but if you and/or your teammates didn’t happen to think of a certain edge-case or even something that might be obvious, then you didn’t write tests for that. Even though the build report might indicate 100% coverage for that piece of code.
I’ve always thought that code coverage is only meaningful when it’s low. Low code coverage can at least definitively tell you that certain parts of code are never executed during test-runs and are thus completely untested. However, it will never definitively tell you which parts of code (or how much of it) are guaranteed to be covered correctly by your tests. As such, i would say that code coverage can only be used as an indication that something is wrong, but never as an indication that something is good/right/correct/great.
October 15th, 2009 at 12:54 pm
Hi Davy,
In my team we have 100% code coverage as a requirement for our build to pass. We don’t for one second think that this means we’ve exhaustively tested our code or that we’re bug free, but we do find it exceptionally useful to find areas that we’ve missed, especially after big refactorings.
As you said, it shows you things that could potentially be wrong, not whether everything is right.
October 15th, 2009 at 12:56 pm
I completely agree. Code coverage is only the “low bar” that your software must clear. Other techniques are required to identify the missed edge cases in your test suite.
If your team is already doing 100% TDD, there is probably no need for code coverage. However, for other (most?) teams, code coverage is a useful indicator of areas that require attention.
October 15th, 2009 at 2:03 pm
Still there is a good reason to use code coverage: find pieces of code you forgot to cover. That means to me that using it occasionally is necessary.
October 15th, 2009 at 8:58 pm
‘Still there is a good reason to use code coverage: find pieces of code you forgot to cover. That means to me that using it occasionally is necessary.’
IMO this should never happen if your doing TDD (ideally BDD) properly – and I don’t mean always writing tests firsts, I mean writing tests based around behaviour. Ideally that behaviour will have come from a use case.
October 15th, 2009 at 9:07 pm
@Awkward Coder
fully agreed
October 16th, 2009 at 9:16 am
[...] There’s Only One Thing You Can Learn From Code Coverage – Davy Brion talks about the concept of code coverage as a metric, and argues that the only meaningful thing that can be derived is that when code coverage is low you have a problem [...]
October 16th, 2009 at 12:16 pm
I completely agree with your post. I would even say that high coverage might even give you a false sense of “confidence” about the fact that your tests are good.
I’d rather be happy with 80% of code coverage but with lot of corner cases test to stress the logic of the application where it’s more likely to break.
October 16th, 2009 at 3:25 pm
I agree that code coverage metrics are better at telling you what isn’t tested than what is, but I think that’s still pretty valuable.
October 16th, 2009 at 3:52 pm
I don’t use code coverage to determine if an application is bug free. I use to determine the every line of code is tested, and that there is a path to every line of code. This is a great way to find dead code. If you are not worried about having every line of coded tested then why even worry about code coverage at all. If you don’t have complete coverage how can you say for certain, that no regression bugs were introduced? It defeats the purpose of automated testing if not all the code gets tested.
October 16th, 2009 at 11:33 pm
@Dave : Bugs can easily be introduced even with 100% code coverage.
I don’t quite understand how tests can identify dead code. If I write a test for something and then remove the call to that code in the app without changing the test, it doesn’t help me at all, in fact it makes it harder to locate the problem. There’s far more usefull tools out there to identify dead code than unit-testing.
On my blog I’ve posted a couple of examples of how you can have 100% code coverage and still have bugs. See the QuickNet preview article, for a simple example.
As far as regression bugs are concerned, again I think it is not the 100% code coverage that guards you against this. Even with full coverage you can have a bug and you need to write an extra test to guard yourself against reintroducing bugs (as demonstrated in aforementioned post). If programming was pure math, then yes, 100% code coverage might do the trick. But there’s side effects (any kind of IO, file, db or other, being a killer) and other unexpected stuff (casts f.i. come to mind) that you need to take into account.
In my experience the effort required to achieve 100% code coverage is better spent in trying to figure out the problem corner cases for code that’s already covered and write tests for those cases, than to put your time into those last couple of percentages which are not going to yield much result.
Sorry about the shameless blog plug, but I thought it might be relevant.
Oh, and if you’re interested in ‘a path to every line of code’, you should check out PEX. It generates the unit tests that cover everything for you. But you’ll still have bugs
October 18th, 2009 at 9:49 am
@Dave
let me throw a modified version of that question back to you:
“If you do have complete coverage, can you say for certain that no regression bugs were introduced?”
I don’t think anyone can truly say yes to that, and that is the whole point of the post
October 19th, 2009 at 1:27 am
Coverage is no guarantee of a bug-free system. Not sure where this idea comes from, but it shouldn’t be too hard to understand. Coverage is *one* of the many indicators of how healthy your system is. It’s *not* the only one. I never heard any coverage proponent advocate otherwise.
Perhaps an analogy is your own physical health. A doctor won’t let you go and promise you eternal life just because your temperature happens to be a nice 98.6 F. Not at all. There are other signs that need to be measured.
In other words, if your coverage/temperature is bad, your system is probably sick. If your coverage/temperature is bad, you need to start measuring other things in addition to coverage/temperature.
From here, and for whatever reason I fail to understand, some people make the argument that it’s useless to measure your body temperature, since a perfect body temperature is not an indicator of perfect health.
October 19th, 2009 at 1:28 am
er:
“In other words, if your coverage/temperature is bad, your system is probably sick. If your coverage/temperature is *GOOD*, you need to start measuring other things in addition to coverage/temperature.”
October 19th, 2009 at 2:36 pm
@awkward: still you can forget to write tests around code (it’s only human to forget). And obviously not all code is behavior driven and described in a use case, so if you never use code coverage, you can only rely on yourself-making-no-mistakes. It’s just an extra check you can do and it may help…
Also code coverage could be a tool for reviewers, to see quickly if everyone done their job properly.