Code Coverage Is A Silver Bullet

Have you ever measured code coverage on any of your projects? How did it work out for you? What problems did it solve? Did it present any new problems? Code coverage is a silver bullet, a silver bullet for understanding how well your tests cover your code. Beyond that, it’s really what you make of it.

What Is Code Coverage

Code coverage is a metric that measures how much of your “production” codebase is being tested by your automated tests, usually unit tests. It’s usually measured at the granularity of a line of code, and sometimes measures methods and classes in aggregate. It’s nothing more than a black and white measurement, how much production code was executed when a given test suite was run. There’s no judgements made to the quality of the tests, simply, how much code did they execute.

Where Code Coverage Gets Tricky

Just to reiterate, code coverage as a metric makes no claims about the quality of tests. If you write a “test” (intentionally in quotes), that simply calls a method and does nothing with the result, the method it calls will have a high level of coverage. That being said, the “test” has done nothing to actually verify an outcome, or craftily provide edge case inducing input. It simply calls the method and discards the result. Voilah, high coverage, crappy test. For example:

class Adder {
  func add(x: Int, y: Int) -> Int {
    return x + y
  }
}

Here’s a corresponding test:

func testAdd() {
  let toTest = Adder()
  toTest.add(2, y: 3)   
}

Adder right now has 100% unit test code coverage! Yay!

Wait a minute, slow down hoss. There’s not a single assertion made in that test. While the test generates a high code coverage metric, it doesn’t validate squat.

TDD FTW

This is where following your test driven development cycle of “Red, Green, Refactor” you’ll never get into this state of test crappiness, and it can truly ensure that your code coverage is a silver bullet. The “red” step of that cycle is critically important, as it ensures that your test actually verifies something. Without knowing that your test can actually fail, you never know that your test actually does anything. And likewise, that your code coverage even means anything.

Be a Professional

The biggest counter argument you will hear about measuring code coverage, is that it can be cheated. Of course it can be cheated! Software “professionals” don’t cheat though. Craftsmen don’t take shortcuts. My life was changed when I read “The Clean Coder” by Bob Martin. He talks about what it takes to be a “professional software engineer.” (He also advocates for 100% code coverage despite all other costs, but I have other opinions on that). It’s a sense of taking pride in your work. With this post, I just want to put this call to action out there, be a professional. Code coverage measurement is just another tool in your toolbox. Code coverage is a silver bullet, but only one that returns to you what you put into it. Crap in, crap out. Use it appropriately, know it’s shortcomings, and know where it shines. When properly used in conjunction with TDD it can powerfully help you continually improve over time. I’d love for you to try it out, and let me know how it works for you.

I was inspired to write this article from listening to episode 67 of “This Agile Life” podcast, where they reference this article, “Is Code Coverage a Silver Bullet?“

Happy cleaning.