Shift left boosts avionics software verification
StoryMarch 20, 2015
Jim Thomas
Test and Verification Solution (TVS)
Many organizations developing avionics software still suffer from poor verification practices. Certain verification techniques can benefit those seeking compliance to avionics certification standards through a strategy of defect avoidance.
Avionics standards encourage a structured approach for software development, but many organizations don’t execute this well. In many cases there is pressure on the software team to start coding in order to meet squeezed time frames, so software requirements and design aren’t fully analyzed and defined, unit testing is skipped, and testing is carried out on a large amount of software. An external company is then paid to produce the documentation and unit tests for the approvals process based on the final software. This situation can arise even if the development has complied with a standard such as DO-178B or DO-178C.
In the end, this approach may find a number of defects, but it typically isn’t that effective – the delivered software has a higher than expected residual number of defects. Inevitably, problems with the initial requirements specification and software design are uncovered during testing; significant numbers of defects found during testing can mean that the software-integration phase takes three or four times longer than expected, or the whole project has to be scrapped and has to start again. All of this means that development costs are much higher than expected and software quality is lower than anticipated.
Improving the verification process brings advantages through shorter development times, dramatically shorter integration times, significantly reduced defects in the software, and a higher quality, on-time delivery.
Avionics standards provide definition
The standards deliberately do not define a particular life cycle or methodology. They define clear objectives and outputs, and these can be produced in different orders, from the planning, requirements, design, coding, integration, configuration management, process assurance, and verification processes.
Rather than producing the outputs as an afterthought, starting with a well thought-out verification plan means that these outputs can be integrated into the development process right from the beginning. Verifying them as they are produced creates a higher-quality product with a much greater chance of success.
Shift left for early verification
One way to do this is to use a technique called shift left. This puts more emphasis on improving the quality of the outputs of the early stages of the development life cycle (the left side of the V-shaped development, see Figure 1), with more rigorous requirements and design stages. This technique is about shifting testing in its widest sense to the left side of the V-model.
Figure 1: The traditional V-shaped software development life cycle.
(Click graphic to zoom by 1.9x)
|
|
Every avionics project starts with the requirements stage, but this is not always done well. The requirements may not be analyzed in sufficient detail – there may be ambiguities in some requirements, inconsistencies sometimes exist between requirements, and the handling of error cases may not be fully addressed. These conditions can lead to problems later in the life cycle.
The key is actually to start with a verification plan, which may seem to be putting the last things first. Verification is much more than just testing the software, however. If requirements and design are more effectively verified by review and analysis, there exists a much more stable base from which to develop the software. Following on from that, thinking about how you will test the source code, what test vectors and stimuli you will use, and from which tools and what environments all helps to construct a coherent and efficient design and verification flow.
One key way to improve the process is to involve experienced testers right at the beginning, in both the requirements and design stages. While avionics experts will put together the requirements, testers can assess the requirements attributes, identifying how testable and traceable the requirements are. The testers will also – and this is critical – highlight ambiguities and inconsistencies. The testers will also be looking at whether failure conditions are fully addressed, an issue that is not usually well covered and which causes significant problems further down the line when an unexpected failure mode emerges in a corner case. Involving testers in the design phase means that the software is designed with testability in mind, features to simplify testing are designed in, and the traceability between requirements and design is assessed. Software that is better-designed from the start can be tested more easily on a host platform rather than having to be tested on the target. Additionally, structural test coverage targets can be more easily met, significantly speeding up test development and the resolution of problems.
When it comes to software testing, shift left also applies. Testing small units of code in isolation provides confidence that they work internally and creates a set of pretested building blocks that go together quickly. A unit may be about 50 to 100 lines of code, so perhaps 20 to 30 units can be integrated together at a time and also tested in isolation. Once tested, these larger software components can then be integrated with other sets of components, and so on. This step-by-step approach to test and integration avoids the difficulties encountered by big-bang integration testing. To be most effective, however, step-by-step testing needs the stable requirements and design base created by better verification earlier in the life cycle.
A verification plan that adopts the shift left concept will result in more predictable time scales and create software with a lower level of defects as defects are avoided or detected earlier on in the life cycle, rather than having to be removed at a later stage.
The future
The new DO-178C specification, approved by the FAA in the U.S. in July 2013, now includes guidance on formal methods. It also clarifies the difference between High Level Requirements, Low Level Requirements, and Derived Requirements and gives a better definition of the exit/entry criteria between systems requirements and system design, allowing the use of high-level models. New tools for model-based engineering use high-level models to generate code and even RTL for silicon automatically.
Formal methods and model-based engineering are consistent with the shift left concept of earlier verification in the development life cycle but they bring different challenges for testing the implemented software.
Avoid defects in the first place
There are ways to implement a verification process that don’t add to the cost of development and still provide a higher quality result. Using a shift left process can significantly enhance the quality of the software for avionics systems. Using independent, experienced testers at the requirements and high-level design stages improves the quality of the requirements and design. This approach also enables the testers to start early with the production of system and integration tests. Developing small units of software that can be tested thoroughly in isolation, and integrating the software in a methodical, step-by-step way, simplifies the integration challenges for complex systems.
Using this process, verification becomes less about defect removal during testing and more about defect avoidance at all stages of the development life cycle. The compliance artefacts that are needed by an assurance process such as DO-178C are produced naturally without having to go back and develop them at the end of the project. The savings in integration time and the improvement in quality reduce costs and enhance reliability, which are key considerations for any software project but particularly for avionics.
Jim Thomas is Director of Software Testing at TVS. He has a PhD in Mathematics from Bristol University and more than 30 years of experience in the software industry with a focus on high-integrity software development.
TVS +44-(0)117-903-1100 www.testandverification.com
