CODE QUALITY BLOG: Is 100 percent code coverage analysis essential?
BlogNovember 19, 2014
Jay Thomas
LDRA Technology
CODE QUALITY BLOG: Safety-critical software standards focus very highly on how to test software effectively. They state that effective software testing requires a disciplined approach in which code coverage is used to provide feedback on the effectiveness of the testing to date. The level of testing rigor applied to a system must be driven by the impact of a system failure. The more significant the consequences, the more rigorous the testing has to be.
Coverage analysis is an essential component in software safety, but two questions then emerge – what coverage – and how do I minimize the effort involved in implementing my process. Let’s take a look at guidance from safety critical process standards and see how they discuss coverage, and how coverage impacts risk management. We will also consider implementation effort – where the basic rule is start simple and build up – and try to understand how these factors come together.
In real life – from a practical point of view in terms of choosing a coverage level, always start with Statement Coverage and work up from there if necessary. Guidance from both DO-178 and ISO 26262 help you determine what coverage level is right for your project. Both standards require a system safety assessment to determine the impact of a failure and the system target failure rate, which in turn defines the level of testing required to prove a system has been tested appropriately. Needless to say, the more significant the impact of a failure, the more rigorous test effectiveness must be. An appropriate level of code coverage is then mandated to prove that an appropriate level of testing has been achieved.
This leads to questions such as how critical to your mission is your system? What kind of failure rate should I target? The following table provides some guidance on selecting an appropriate level of coverage as discussed by the Federal Aviation Administration (FAA) with respect to DO-178.
Code coverage as a measurement of testing rigor must be applied carefully. For instance, it is not appropriate to measure the coverage achieved by executing the system without a test plan. Execution must be driven by test planning and requirements. Generally, the guidance from the safety-critical software standards is that to prove an appropriate level of testing rigor, testing must be requirements driven and performed at the system level. However, with appropriate requirements, you can supplement this testing with tests driven at the unit level. Only then is it appropriate to use coverage analysis to measure test completeness.
In practice, achieving 100 percent code coverage from system level testing is neither appropriate nor necessary. Achieving the maximum code coverage for your system is an iterative process. Using code coverage results as feedback, it’s possible to identify deficiencies in the testing process such as missing requirements, missing test cases, unreachable, unneeded, or dead/deactivated code. Test cases can then be added, requirements addressed, and code refactored to address the issues raised. The testing can then be updated and repeated until the test effectiveness objectives for your project have been met. This may include accounting for unused code (e.g. when only part of an open source component is used) or augmenting the system level test results with results from a test harness, or even code inspection.
When choosing a tool to help with coverage measurements, it is important to note that not all coverage analysis tools are created equal, and choosing the wrong tool can compromise your ability to accurately measure coverage—or worse—provide incorrect results. Here are some issues to consider when selecting a coverage analysis tool:
• What’s the memory footprint of the coverage measurement implementation, particularly if you’re testing an embedded system?
• Is your embedded system supported by the tool?
• What’s the memory footprint of the run-time data? Does your system have enough memory to make meaningful measurements?
• Will the instrumentation affect the system run-time behavior?
DO-178 provides guidance on these decisions by requiring that any tool used for measuring code coverage must be verified to produce accurate, reliable results in the target environment. You need to therefore make sure the tool you pick has been qualified for DO-178, so that the results it produces can be used with confidence and without further verification. Check your tool’s pedigree.
Code coverage — delivering that essential assurance
The code quality for any software project can benefit from the application of a few simple guidelines from safety-critical standards. To control test effectiveness, the impact of testing must be measured using code coverage, using a code coverage level that is appropriate for the testing rigor required for the software. To ensure an appropriate level of testing rigor, all testing must then be requirements-based and performed at the system level. Test, measure, repeat. The feedback, knowledge, and understanding required to improve test effectiveness is simply not possible without code coverage analysis. When choosing a coverage analysis tool, ensure that you choose a DO-178 qualifiable tool to ensure that you select a tool of an appropriate pedigree. By following these guidelines, any software project can achieve the levels of software quality normally expected of safety-critical systems.
