<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=443149619225659&ev=PageView&noscript=1"> Chasing the Elusive 100% - A New Way of Measuring Code Coverage

We would like to email you valuable insights on software development and DevOps! (We promise not to stalk you or share your info)

 

Chasing the Elusive 100% - A New Way of Measuring Code Coverage

Chasing the Elusive 100% - A New Way of Measuring Code Coverage

When it comes to measuring unit test code coverage, ask 100 developers for the ideal target percentage and you’ll likely get 100 different, and equally justifiable, answers. That’s because unit testing can be very dependent on the type of project and the way the code has been written.

Code coverage is a controversial metric because in some organizations teams are judged on how high a percentage they have. Other companies even have processes in place that reject source code commits if the new code causes the code coverage to drop below a certain threshold. Developers object to this measurement because code coverage doesn’t tell you the quality or the effectiveness of the tests, which is to make sure that changes in code do not break existing logic.

What is code coverage?

Before we dive into percentages and how to apply the standard, let’s back up and define what code coverage is. Code coverage is a metric that shows teams how much code in a given project is currently being evaluated by unit tests. It is calculated by dividing the number of lines that were run with your test suite by the number of total lines of code. The resulting number is your code coverage percentage. In addition to the percentage, you can also see reports of exactly which code isn’t being tested.

Determining the ideal coverage target

So how can teams reliably use code coverage percentages in a way that meets everyone’s needs? The secret lies in how we define the target.

There’s a popular quote that has been attributed to Albert Einstein, if he had only one hour to save the world, he would spend 55 minutes defining the problem, and five minutes on the solution. While it wouldn’t make for a good action movie plot, that logic suggests that by developing a clear understanding of exactly what the measurement needs to accomplish, you’re that much more likely to achieve the desired result.

Typically, developers advise targeting 80% coverage for a project’s unit tests. That percentage feels reasonable at first. But dig a little deeper, and you’ll quickly find that 80%—or 60% or 90%, for that matter—doesn’t take into account all the code that doesn’t make sense to be tested in the first place. On a given project, it’s fair to presume that 20% of the code doesn’t require unit testing, either because its core function is so minimally impactful that no test is needed, or due to the time it would take to write and execute the test outweighs the benefit of testing that segment of code in the first place (such as, third-party frameworks, live web services, or simple auto-generated properties with no business logic, for example). When we take these unknowns into account, suddenly the magic 80% rule falls flat. Is that 80% the right code to be tested? How do we determine if some of our important code was excluded?

Even thinking about it gets complicated.

My conclusion

In my experience, teams are better served by selectively identifying which code to test and which code to exclude from code coverage metrics by marking certain code as excluded from analysis. Once that universe is defined, you can set the target percentage to 100% and use code coverage reports to show what code hasn’t been reviewed for unit tests. This changes the code coverage reports from a metric to a discussion of whether unit tests would be valuable for the newly added code. In this way, instead of chasing an arbitrary metric, teams can use code coverage reports to make their code and their unit tests more effective.

 

The Challenges of Agile Software Development eBook

Recent posts

Effective Digital Transformation Begins with Identifying Business Values
Digital Roadmapping: What is it and why is it important?
2018 in Review - Top 3 Blogs of the Year
People Are at the Heart of Projects
The Path to Tech Entrepreneurship: An Interview with Carl Turner

 

Share this article

 

About the author

With a background in business and as a weekend studio potter, Brian White’s unique experiences fuse his creative passion with technical problem solving, bringing software solutions into form. As a senior solutions architect, he questions the status quo and asks if there’s a better way to do something. Brian is a self-taught developer who has worked on projects across many industries and technologies since 2001. He has juggled many roles as a full stack developer - having a hand in everything from requirements to architecture, testing to deployment. Since moving to Small Footprint in 2012, Brian has blended his attention to detail from the developer-side with a keen understanding of the business and technology needs of his clients.
Ready to partner with an expert in custom software development? Learn about our fast, cost-effective approach to creating world-class software solutions.
301 N Main St, Winston Tower, Suite 2206
Winston-Salem
NC
27101
USA
Republicii 24
400015 Cluj
Romania

The Small Footprint Blog

Keeping up with the latest in custom software development? Visit Small Footprint's blog for expertise, insights and innovative software strategies.