Enhancing military and aerospace software testing with data-driven development strategies

Story

May 16, 2024

When organizations in the military and aerospace sectors adopt a data-driven development (DDD) process, it becomes crucial that they select key performance indicators (KPIs) that are relevant to the industry’s unique challenges and requirements. For software developers and lead engineers in this domain, essential raw KPIs may include tracking the number of requirements implemented per development cycle and the number of lines of code written per day per engineer. These metrics, often recategorized as velocity and sprint burndown, enable valuable insights into labor costs and time-to-market considerations.

Additionally, other software development KPIs pertinent to the military and aerospace industry may encompass code complexity, code execution time, memory usage, number of threads, and others aimed at helping development teams adapt to evolving requirements, prioritize issues, and address high-risk areas efficiently.

For the quality assurance (QA) teams operating within the military and aerospace sector, collecting KPIs such as the average number of defects per line of code becomes paramount. Typically, embedded software in this industry maintains an average of 15 to 50 issues per 1,000 lines, serving as a critical metric for evaluating software robustness and determining whether the application is fit for deployment or delivery.

An 85% defect removal efficiency (DRE) is generally considered acceptable for average nonsafety applications; however, for embedded safety-critical software used in military and aerospace applications, the acceptable range rises substantially to between 95% and 99%. Achieving a DRE of 99.9% remains an aspiration rather than a practical reality.

Notably, projects in the military and aerospace industry that adopt comprehensive testing methodologies – including static analysis as the initial stage of testing, followed by manual peer reviews of key features and a series of multistage testing phases such as unit testing, integration testing, system testing, regression testing, and code coverage – report achieving DRE rates within the desired range of 95% to 99%. Moreover, these teams experience tangible business benefits such as shortened development schedules, reduced project failures, fewer field problem reports, and lower labor costs. (Figure 1.)

[Figure 1 ǀ Incorporating data-driven development (DDD) practices throughout projects, from planning to deployment, empowers software engineers to improve decision-making, performance, and quality, ultimately delivering increased value to customers and stakeholders. When this approach is integrated into the DevOps methodology, it becomes known as DataOps.]

Applications in the military and aerospace domain

In the aerospace sector, data-driven approaches are integral across a spectrum of applications and are aimed at enhancing software design, testing, and deployment processes. For instance, modern aircraft equipped with advanced flight-control systems utilize real-time data from an array of sensors – such as radar, lidar, and cameras – to enable features like autopilot and collision avoidance. Telematics systems further gather performance data, operational parameters, and user interactions, with the goal of fostering continuous enhancements in software quality and safety.

Expanding beyond these applications, key areas in the military and aerospace domain include:

• Mission planning and execution: The use of data analytics and potentially quantum computing in optimizing mission planning and execution involves the development of software systems to process and analyze historical mission data, environmental conditions, and performance metrics. This approach likely involves creating algorithms, software tools, and systems to handle and analyze large volumes of data, as well as implementing decision-making processes based on this analysis.
• Aircraft and vehicle systems: Software plays a crucial role in analyzing sensor data, monitoring equipment performance, and predicting maintenance needs in aerospace and military vehicle systems. These needs involve developing software for data collection, analysis, and visualization, as well as implementing predictive-maintenance algorithms.
• Surveillance and reconnaissance: Analyzing data collected from surveillance systems involves software development to process and analyze large volumes of data from drones, satellites, and ground sensors. These steps likely involve developing software for image processing, pattern recognition, and data visualization.
• Simulation and training: Military-training simulations rely heavily on software development to create realistic scenarios and adapt to user behavior. Such efforts involve developing simulation software, incorporating machine learning algorithms for adaptive training, and analyzing user data to improve training effectiveness.
• Cybersecurity: Developing software for analyzing network traffic, detecting anomalies, and identifying security threats involves software development in cybersecurity applications. Development of cybersecurity software includes developing algorithms for anomaly detection, implementing machine learning models for threat detection, and developing software tools for network monitoring and analysis.

In the aerospace industry, real-world data serves as invaluable input for various testing activities, including unit testing, regression testing, performance testing, error handling, security testing, and test case design. By simulating diverse operating conditions and user scenarios, engineers and testers can create comprehensive test cases driven by datasets, enabling exhaustive testing of software functionalities under different circumstances.

Moreover, recognizing the significant costs and time investments required to ensure that software developed for these diverse applications is not only reliable but – crucially – safe and secure, one might inquire about the most efficient and effective approach to software testing. In essence, the answer lies in integrating testing seamlessly into the software-development process.

Employing CI/CD and data-driven testing

Continuous integration and continuous delivery (CI/CD) combine to form a robust software-development practice known as frequent integration, merging smaller build units into cohesive apps, services, libraries, or components, alongside continuous delivery or deployment. The core objective is to automate software testing to promptly identify build and integration issues and errors. By incorporating DDD principles into the continuous integration workflow, the industry can attain elevated levels of software quality and reliability. (Figure 2.)

[Figure 2 ǀ Shown: Data gathering and visualization of software quality workflow.]

In a typical workflow, developers create branches for their work, write new code, or modify existing code, then submit pull requests for review. Within the CI/CD pipeline, automated test scripts – including static analysis, unit testing, and code coverage analysis – are executed for each code commit. Static analysis is actually a foundational step in enhancing software quality.

For C and C++, adhering to coding standards like MISRA C 2023 and MISRA C++ 2023, coupled with SEI CERT C or CERT C++ security standards in conjunction with MISRA, ensures the development of the safest and most secure applications. Additionally, leveraging containerization technologies such as Docker simplifies the testing process, ensuring thorough validation of software changes before integration into the main codebase.

Platforms like GitHub, coupled with self-hosted runners, are prevalent in the avionics and aerospace industries for hosting CI/CD pipelines. These runners, whether physical servers, virtual machines, or container images, seamlessly integrate data-driven tests into the development workflow, resulting in software that meets stringent safety, security, and regulatory requirements while aligning with business objectives.

Continuous integration and delivery have become standard practices in embedded development. Transitioning from a waterfall process to CI/CD and Agile development yields significant benefits in risk reduction and enhancements in quality and security. Embedded developers prioritize security, and CI/CD facilitates DevSecOps [development, security, and operations] by embedding security requirements and controls throughout the pipeline.

Containers are inherently compatible with CI/CD, supporting rapid deployment and portability across diverse host environments with robust versioning and centralized control. Containerized development environments are essential for secure development within a DevSecOps pipeline, providing a reproducible application environment with integrated security controls.

Continuous testing is indispensable in a well-functioning CI/CD pipeline, as testing consumes the most time and resources. By shifting testing earlier in the development life cycle, continuous testing frameworks streamline processes. Automation and targeted focus on high-risk areas enable testing to become less of an impediment in continuous processes. Tools supporting automation and optimization are critical for continuous testing and can drive larger code coverage, intelligent test execution, and bidirectional traceability, thereby further enhancing software quality and reliability.

Ricardo Camacho is director of Safety & Security Compliance at Parasoft. He has decades of experience in systems and software engineering of real-time safety- and security-critical systems for various industries. His career has spanned multiple roles, including technical product marketing, project management, solution architect/technical sales, and embedded software and systems engineering, which he has performed at companies including IBM, Xerox, Vector, and GE Rail.

Parasoft     https://www.parasoft.com/