CODE QUALITY BLOG: Is code coverage analysis necessary?

Blog

October 20, 2014

Jay Thomas

LDRA Technology

CODE QUALITY BLOG: When programmers discuss testing, a question that often comes up is how much is enough? Do I need to get 100 percent coverage? Let’s first address the elephant in the room: it’s impossible to fully test a piece of software. Don’t believe me? A 100-line program described by Glennford Myers in the 1976 book "Software Reliability: Principles and Practices" had as many as 1018 unique paths. In reality when you realize that contemporary software often exceeds hundreds of thousands, if not millions of lines of code and you quickly realize how impractical it is to completely test a piece of software.

I was once on vacation and received a phone call from a satellite phone in a remote part of the world to let me know that the mission that I was working on was “in the drink” – a euphemism indicating the vehicle which was expected to fly was under water. In the end, it turned out to be a failure due to a rusty nut, but in the process of investigating that failure, we discovered many paths through our safety-critical code which were not covered by our testing regime. While all that was worked out in the months that followed, my vacation was difficult to enjoy as I didn’t know what happened.

So how much software testing is enough? This is a question addressed by mission- and safety-critical software projects, and there are some valuable lessons to be learned from them. At its most basic, their approach is test, measure, repeat. Like Tom DeMarco quipped “You can’t control what you can’t measure”, and the measurement referred to here is Structural Coverage Analysis, or code coverage. Many industry standards (e.g. DO-178 for aerospace and ISO 26262 for automotive) require the use of code coverage as the only means of proving test completeness.

Structural Coverage Analysis is a measure of the code structures exercised by a given test or set of tests. One of the most basic metrics, Statement Coverage (SC), measures the code statements executed by testing. A more rigorous measurement is Decision Coverage (DC) which includes the objectives of SC plus measuring which decision paths have been taken from each decision point in code. More rigorous still is Modified Condition/Decision Coverage (MC/DC) which includes the objectives of DC plus measuring which values of each of the conditions in a decision point have been taken. All three of these measurements are mandated by the DO-178 standard.

Code coverage is normally expressed as a percentage and the more thoroughly tested a piece of software is, the higher the code coverage percentage and the smaller the chance that the software contains defects.

Safety-critical software standards focus very highly on how to test software effectively. They state that effective software testing requires a disciplined approach in which code coverage is used to provide feedback on the effectiveness of the testing to date. The level of testing rigor applied to a system must be driven by the impact of a system failure. The more significant the consequences, the more rigorous the testing has to be.

A word of caution, however; 100 percent coverage does NOT necessarily translate to effective testing. Testing with the objective of maximizing code coverage can miss significant issues in code that rigorous testing helps to identify, such as:

• Requirements related errors
• Missing code
• Data flow-related bugs
• Race conditions
• Hardware compatibility/configuration

Nevertheless, as a measure of testing completeness and rigor, code coverage is unsurpassed and should be adopted by every software development project.