Trusting the tools: An agile approach to tool qualification for DO-178C

Story

March 07, 2012

The new avionics software safety standard DO-178C, along with its supplemental Software Tool Qualification Considerations (DO-330), has clarified and expanded the tool qualification guidance provided in DO-178B. The challenge of maintaining qualification-ready tools throughout a system?s evolution can be expedited through an approach based on agile development principles.

If a manual activity required for avionics software certification is reduced or replaced by an automated tool, and the output of that activity is used without being verified, then the developer needs to qualify the tool: demonstrate that the tool is at least as trustworthy as the activity that it is replacing. The new avionics safety standard, DO-178C – together with its companion Software Tool Qualification Considerations, DO-330 – has clarified and expanded the tool qualification guidance defined in DO-178B. The following discussion summarizes the new guidance and describes an agile approach to maintaining qualification-ready tools in the presence of system maintenance and changes.

Tool qualification in DO-178B

DO-178B[1], a commercial avionics software safety standard that is finding increasing usage in military aircraft development, is often referred to as “process based”: It specifies an interrelated collection of software life-cycle processes, each comprising a set of activities and associated objectives. The activities produce outputs (“artifacts”) that are evaluated by certification authority personnel to see if they comply with the objectives specified in DO-178B. The applicable objectives (and thus the applicable activities and artifacts) depend on the Software Level: the criticality of the software in ensuring aircraft and occupant safety. The levels range from E (no effect) to A (software failure can directly lead to loss of aircraft and, therefore, lives).

Some DO-178B activities are automatable, and the standard describes how a tool can be trusted to replace or reduce a manual activity if the tool’s output is used without being verified. It defines two categories: development tools and verification tools. A development tool generates output that is part of the airborne software and thus has the potential to introduce errors. An example is a code generator that produces source code from a model-based design. A verification tool cannot introduce any errors but may fail to detect errors, for example, a static analysis tool that identifies variables that are read before being initialized.

Tool qualification entails preparing, among other data items, the Tool Operational Requirements (TOR). The TOR defines various properties of the tool including its features, installation, usage, and operational environment.

A development tool needs to be qualified if, and only if, the software generated by the tool will not be subjected to the same applicable certification objectives as the other airborne software. Development tool qualification entails meeting the same objectives as for the certification of the airborne software. (Although compilers and linkers are development tools, qualification is not required since their output is verified through other DO-178B activities. Indeed, qualification would be expensive and would not simplify the effort in meeting other objectives such as traceability analysis.)

Qualifying a verification tool is considerably simpler than qualifying a development tool, in part because DO-178B’s philosophy is to encourage the use of such tools to automate activities involving repetitive and rule-based tasks, which are better performed by automated tools than by humans. Qualifying a verification tool basically consists in demonstrating that the tool complies with its TOR.

Tool qualification in DO-178C

Tool qualification has been an important part of DO-178B certification, but several issues have arisen in practice:

The distinction between a verification tool and a development tool is not always straightforward. Moreover, a verification tool might not simply automate a specific activity; its output may also be used to eliminate or reduce some other activity.
Requiring a development tool to meet the same objectives as the airborne software is unnecessarily restrictive, since the operational environments are different. For example, an unbounded recursion in the avionics software could exhaust stack storage and lead to a system failure; the same behavior in a development tool would not present a safety hazard.
Although tool qualification is intrinsically in the context of a specific system, it would be beneficial if the qualification requirements expedited reuse of qualified tools on a modified version of an existing system.

All of these issues are addressed in either DO-178C[2] or its accompanying supplement DO-330, Software Tool Qualification Considerations[3].

The terms “development tool” and “verification tool” have been replaced by three criteria. Criterion 1 corresponds to a development tool (that is, the tool could insert an error into airborne software). Criterion 2 corresponds to a verification tool that could fail to detect an error and is used to reduce other development or verification activities. Criterion 3 corresponds to a verification tool that could fail to detect an error but is not used to reduce other development or verification activities.
The required qualification for a tool – its Tool Qualification Level (TQL) – depends on its Criterion and on the Software Level of the software that the tool is used for, as shown in Table 1. The TQL ranges from 5 (comparable to a DO-178B verification tool) to 1 (similar to Software Level A). The activities and data items associated with each TQL are defined in a separate document, DO-330, with the same structure as DO-178C. DO-330 provides comprehensive guidance for tool qualification and recognizes the differences between the execution environments for the airborne software and the tool.
DO-330 explicitly covers the usage of previously qualified tools. In brief, the reuse of a previously qualified tool is allowed as long as the developer can demonstrate, through a change impact analysis, that the tool still complies with its TQL requirements despite any changes in the operational environment or to the tool itself.

Table 1: The required qualification for a tool – its Tool Qualification Level (TQL) – depends on its Criterion and on the Software Level of the software for which the tool is used.

(Click graphic to zoom by 1.9x)

Reuse of previously qualified tools

The ability to reuse, or easily adapt, the qualification artifacts for a previously qualified tool is especially important. DO-178B provided no explicit guidance here. Tool qualification that was performed for one system would need to be repeated for any new system or if any aspect of the tool or environment changed. As a result, a project manager would commonly choose the operational environment and tools at an early stage, and then commit to these versions so that the tool qualification artifacts could be used during final system certification. This is sometimes referred to as the “big freeze,” where the environment and tools are locked in early.

DO-330 addresses these issues. Specific guidance for previously qualified tools allows reuse of the qualification artifacts as long as nothing has changed that would affect qualification. It considers three scenarios:

Reuse of a previously qualified tool without change – An example is when a tool is used for related projects or on multiple phases of an existing project. The developer needs to identify the approach and rationale in the plans.
Changes to the tool operational environment – The developer needs to update one or more of the plans, but the bulk of the original qualification artifacts may be reused as is. Only the updated artifacts related to the operational environment need to be reviewed by the certification authority.
Changes to the tool itself – A change impact analysis has to be provided, but tool requalification still has a reduced cost, essentially only requiring activities associated with aspects that have changed or are affected by the change. The key is to be able to exactly determine and specify what has changed and what these changes impact, or perhaps more importantly, what they do not impact.

Agile requalification

Based on the tool qualification guidance – either from DO-178B or from DO-178C and DO-330 – it is possible to define a framework for tracking the changes to a tool or its operational environment and for automatically initiating the tool qualification activities triggered by the changes.

For example, a tool can be initially developed and qualified based on the objectives defined in DO-178C and DO-330. The full tool development life-cycle processes and their associated qualification artifacts can be captured and maintained in a Configuration Management (CM) system, including all dependence relationships (see Figure 1). The core CM system allows basic regeneration of all qualification data and artifacts needed to reproduce a tool qualification. The full structure allows impact and change analysis. In this way any change to the tool’s operational environment or to the tool itself can be tracked. Most importantly, the structure will clearly show which parts of the tool and its artifacts are not affected and thus can remain unchanged and retain their previous review and qualification readiness.

Figure 1: The full tool development life-cycle processes and their associated qualification artifacts can be captured and maintained in a Configuration Management (CM) system, including all dependence relationships.

(Click graphic to zoom by 1.9x)

Transitioning to the new qualification guidance

DO-178B is effectively a subset of DO-178C. Thus, a project can continue with the development and certification plans established for DO-178B while migrating chosen portions to DO-178C, for example, to exploit the tool qualification objectives in DO-330. Therefore, both existing DO-178B projects and new DO-178C projects can take advantage of DO-330’s cost-effective guidance on tool qualification and requalification.

The AdaCore Qualifying Machine framework[4], an in-progress implementation of the agile technique described in the previous section, supports this approach. It can help projects avoid the “big freeze,” so that tools and development environments can evolve smoothly. Tools may be upgraded to newer versions as updates become available, without the risk of losing the tool qualification required for system certification.

References:

[1] RTCA SC-167/EUROCAE WG-12. RTCA/DO-178B – Software Considerations in Airborne Systems and Equipment Certification, December 1992.

[2] RTCA/DO-178C – Software Considerations in Airborne Systems and Equipment Certification; publication expected in 2012.

[3] RTCA/DO-330 – Software Tool Qualification Considerations; publication expected in 2012.

[4] www.open-do.org/projects/qualifying-machine

Dr. Benjamin Brosgol is a senior member of the technical staff at AdaCore. He has more than 30 years of experience in the software industry, concentrating on languages and technologies for high-integrity systems. He has presented papers and tutorials on safety and security certification at numerous conferences and has published articles on this subject in a variety of technical journals. He holds a Ph.D. in Applied Mathematics from Harvard University. He can be contacted at [email protected].

Greg Gicca is Director of Safety and Security Product Marketing at AdaCore. He has more than 20 years of experience in designing and implementing software development tools and has participated in industry and government groups responsible for defining software quality evaluation standards. He has concentrated on the safety and security arena for embedded systems, with a particular focus on the DO-178B safety standard and the Multiple Independent Levels of Security (MILS) architecture. He can be contacted at [email protected].

AdaCore 212-620-7300 www.adacore.com www.linkedin.com/company/adacore www.twitter.com/AdaCoreCompany