Comments on: There’s Only One Thing You Can Learn From Code Coverage

By: Koen

Koen — Mon, 19 Oct 2009 12:36:33 +0000

@awkward: still you can forget to write tests around code (it’s only human to forget). And obviously not all code is behavior driven and described in a use case, so if you never use code coverage, you can only rely on yourself-making-no-mistakes. It’s just an extra check you can do and it may help…

Also code coverage could be a tool for reviewers, to see quickly if everyone done their job properly.

By: Julio Santos

Julio Santos — Sun, 18 Oct 2009 23:28:36 +0000

er:

“In other words, if your coverage/temperature is bad, your system is probably sick. If your coverage/temperature is *GOOD*, you need to start measuring other things in addition to coverage/temperature.”

By: Julio Santos

Julio Santos — Sun, 18 Oct 2009 23:27:19 +0000

Coverage is no guarantee of a bug-free system. Not sure where this idea comes from, but it shouldn’t be too hard to understand. Coverage is *one* of the many indicators of how healthy your system is. It’s *not* the only one. I never heard any coverage proponent advocate otherwise.

Perhaps an analogy is your own physical health. A doctor won’t let you go and promise you eternal life just because your temperature happens to be a nice 98.6 F. Not at all. There are other signs that need to be measured.

In other words, if your coverage/temperature is bad, your system is probably sick. If your coverage/temperature is bad, you need to start measuring other things in addition to coverage/temperature.

From here, and for whatever reason I fail to understand, some people make the argument that it’s useless to measure your body temperature, since a perfect body temperature is not an indicator of perfect health.

By: Davy Brion

Davy Brion — Sun, 18 Oct 2009 07:49:39 +0000

@Dave

let me throw a modified version of that question back to you:
“If you do have complete coverage, can you say for certain that no regression bugs were introduced?”

I don’t think anyone can truly say yes to that, and that is the whole point of the post

By: kilfour

kilfour — Fri, 16 Oct 2009 21:33:20 +0000

@Dave : Bugs can easily be introduced even with 100% code coverage.
I don’t quite understand how tests can identify dead code. If I write a test for something and then remove the call to that code in the app without changing the test, it doesn’t help me at all, in fact it makes it harder to locate the problem. There’s far more usefull tools out there to identify dead code than unit-testing.
On my blog I’ve posted a couple of examples of how you can have 100% code coverage and still have bugs. See the QuickNet preview article, for a simple example.
As far as regression bugs are concerned, again I think it is not the 100% code coverage that guards you against this. Even with full coverage you can have a bug and you need to write an extra test to guard yourself against reintroducing bugs (as demonstrated in aforementioned post). If programming was pure math, then yes, 100% code coverage might do the trick. But there’s side effects (any kind of IO, file, db or other, being a killer) and other unexpected stuff (casts f.i. come to mind) that you need to take into account.
In my experience the effort required to achieve 100% code coverage is better spent in trying to figure out the problem corner cases for code that’s already covered and write tests for those cases, than to put your time into those last couple of percentages which are not going to yield much result.

Sorry about the shameless blog plug, but I thought it might be relevant.

Oh, and if you’re interested in ‘a path to every line of code’, you should check out PEX. It generates the unit tests that cover everything for you. But you’ll still have bugs

By: Dave

Dave — Fri, 16 Oct 2009 13:52:25 +0000

I don’t use code coverage to determine if an application is bug free. I use to determine the every line of code is tested, and that there is a path to every line of code. This is a great way to find dead code. If you are not worried about having every line of coded tested then why even worry about code coverage at all. If you don’t have complete coverage how can you say for certain, that no regression bugs were introduced? It defeats the purpose of automated testing if not all the code gets tested.

By: nappisite

nappisite — Fri, 16 Oct 2009 13:25:53 +0000

I agree that code coverage metrics are better at telling you what isn’t tested than what is, but I think that’s still pretty valuable.

By: Stefano Ricciardi

Stefano Ricciardi — Fri, 16 Oct 2009 10:16:24 +0000

I completely agree with your post. I would even say that high coverage might even give you a false sense of “confidence” about the fact that your tests are good.

I’d rather be happy with 80% of code coverage but with lot of corner cases test to stress the logic of the application where it’s more likely to break.

By: Reflective Perspective - Chris Alcock » The Morning Brew #456

Reflective Perspective - Chris Alcock » The Morning Brew #456 — Fri, 16 Oct 2009 07:16:40 +0000

[...] There’s Only One Thing You Can Learn From Code Coverage – Davy Brion talks about the concept of code coverage as a metric, and argues that the only meaningful thing that can be derived is that when code coverage is low you have a problem [...]

By: Davy Brion

Davy Brion — Thu, 15 Oct 2009 19:07:12 +0000

@Awkward Coder

fully agreed

By: Awkward Coder

Awkward Coder — Thu, 15 Oct 2009 18:58:01 +0000

‘Still there is a good reason to use code coverage: find pieces of code you forgot to cover. That means to me that using it occasionally is necessary.’

IMO this should never happen if your doing TDD (ideally BDD) properly – and I don’t mean always writing tests firsts, I mean writing tests based around behaviour. Ideally that behaviour will have come from a use case.

By: Koen

Koen — Thu, 15 Oct 2009 12:03:03 +0000

Still there is a good reason to use code coverage: find pieces of code you forgot to cover. That means to me that using it occasionally is necessary.

By: Akash Chopra

Akash Chopra — Thu, 15 Oct 2009 10:56:17 +0000

I completely agree. Code coverage is only the “low bar” that your software must clear. Other techniques are required to identify the missed edge cases in your test suite.

If your team is already doing 100% TDD, there is probably no need for code coverage. However, for other (most?) teams, code coverage is a useful indicator of areas that require attention.

By: David

David — Thu, 15 Oct 2009 10:54:54 +0000

Hi Davy,

In my team we have 100% code coverage as a requirement for our build to pass. We don’t for one second think that this means we’ve exhaustively tested our code or that we’re bug free, but we do find it exceptionally useful to find areas that we’ve missed, especially after big refactorings.

As you said, it shows you things that could potentially be wrong, not whether everything is right.