How much and what kind of coverage analysis is enough?

Blog

November 21, 2016

CODE QUALITY BLOG: The discussion of coverage analysis can bring up a number of different assumptions that do not always agree. Does it mean checking that all the code has been executed? Does it mean that all requirements have been exercised and tested? Does it bring up some number other than 100 percent that can be relied upon for functioning code? What we want to do is to assure ourselves that a program has been thoroughly tested to the point it can be relied upon, even in life-threatening situations. How do we achieve this and what aspect of ?coverage? will allow us to rest easy?

Software testing and analysis can be thought of as a holistic activity that is made up of a number of interdependent parts. These include such things as requirements tracing, static and dynamic analysis, coding standards compliance and more—including coverage analysis. At the end of the day, coverage analysis is supposed to give us a sense of how well and how thoroughly a piece of code has been tested. This, of course, depends on how well and thoroughly the other testing methodologies have been applied and their results. So it is in effect a testing of our testing, not a test of the program itself.

So what is it, then, that can give us a good sense of how well we’ve tested?

One approach might be to check to see that all lines in the program have been executed. However, that alone does not tell us how the path of execution got to those lines or in what order and under what conditions it did so. And it does not directly relate back to the requirements. The requirements are, after all, the basis from which testing—both automatic and manual—is generated in the first place.

Another take at coverage would be branch coverage, which shows the paths of execution between sections of code, but not necessarily every line. Branch coverage can reveal the structure of a program in terms of the path of execution. A branch is a “this” or a “that.” It tells us which way the execution can go, but it does not say why the code would go one way or another. This gives us a picture of the structure of execution, but even if it reveals that all the branches were exercised at least once during execution, it does not show the conditions under which one or the other path was taken from a branch. That is, it does not necessarily indicate that all the circumstances (Boolean expressions, conditions) were tested or at least all that would satisfy the requirements.

The expression If A is a branch. It could, of course, be a more complex expression that would result in a true or false A, so the resulting value of A is the decision. Decision coverage means that every point branch has been invoked at least once and all of the decisions each branch takes have been executed at least once. This is a stronger measure than branch coverage as it links the branches to paths. Testing designed to execute each outcome of each decision point in a program, then, is branch decision testing. However, the execution of each outcome does not deal with the different inputs and conditions that could lead to that (if,then) decision. For that we must turn to branch/decision testing and its cousin, modified condition/decision coverage (MC/DC).

MC/DC invokes each point of entry and exit in the program at least once using every condition so that a decision has taken all possible outcomes at least once, and that changing any condition in a decision can be shown to independently affect that decision. A condition is shown to independently affect a decision's outcome by varying just that condition while holding fixed all other possible conditions.

While metrics are great – metrics alone do not help us rest assured our code will work the way we expect. Testing must be done in relation to the program’s requirements – does the program do what it is supposed to do – and these tests must be the ones that generate and trace to the appropriate coverage metrics. This view – enhancing coverage with traceability – is the key to functional safety described by diverse standards such as DO-178B and IEC 61508. This combination allows us to know the code does what it is supposed to – and we have executed it through test scenarios.

Thus this is the view that allows us to sleep at night – rest assured our software will work the way we want.