Effective use of open source in embedded applications

Story

February 17, 2011

Epsen Boch

Galleon Embedded Computing

Very specialized and often in-house designed solutions using highly proprietary hardware and software have traditionally dominated the embedded industry. The massive focus on cost reductions and the increased use of COTS products have led to lower hardware costs, but the cost of software remains high. However, if used and managed in the right way, open source software projects can help Military and Aerospace (M&A) system developers complete projects on time and within budget while improving the overall quality and stability of the resulting product.

The military embedded industry is in the middle of a massive change of implementation strategy. For many years, military programs were more or less exempt from commercial consideration. Both politicians and the public in general have set focus on the cost of military programs. Moore’s law and increased use of COTS products have contributed to lower hardware cost, but software has not experienced a similar cost reduction path.

Can open source software help in getting more out of limited project budgets? According to a recently released VDC report on embedded operating systems[1], the use of Linux in embedded projects shows an annual growth rate of about 50 percent, while new projects using non-Linux operating systems show a steady state. The most interesting observation is that among the projects using Linux, about 80 percent use free public Linux distributions. However, the study also shows that this 80 percent includes the projects with the longest implementation times, indicating that the use of free software does not necessarily reduce the total project cost. Thus, open source projects must be properly planned and managed. When the benefits of open source software are utilized, the project not only completes on time and within budget, but also results in a more robust and flexible product. The challenges and benefits of open source in M&A applications are discussed, and an application example is presented.

Free open source versus commercial distributions

Even though the software is open source, it does not necessarily have to be free. What are the benefits of choosing a commercial open source distribution over a free alternative? The commercial Linux providers argue for better support, an integrated tool chain, and feature enhancements. Are they right?

Support is a major concern in all projects – and the most compelling argument for choosing a commercial solution. Most organizations have experienced poor support and recognize the cost of this. Unfortunately, even for commercial alternatives, responsiveness is not always a given. It is often hard to get access to the development engineers. The best one can hope for is for the issue to be resolved in the next official release, which can be months away. For free distributions with an active user group, support questions most often get responses immediately with initial suggestions to workarounds and fixes. Oftentimes a fix is available within hours.

The next argument is: “Commercial alternatives offer an integrated tool chain with cross-compilers, ICE, debuggers, memory management tools, and so on.” This is mostly true, but the number of free alternatives increases every day, and the quality of the tools continues to improve.

The area where commercial Linux distributions have the most to offer is feature enhancements. No community-based Linux distributions offer good real-time support currently. Further, DO-178 safety certification, EAL Common Criteria, and MILS are all examples of requirements not likely to be implemented as part of a free Linux distribution. In fact, it is not even clear that any Linux distribution will ever meet these most stringent criteria. For applications targeting these standards, the commercial offerings (Linux or hard RTOS) will continue to be the best bet.

Challenges in open source projects

Given that the choice has been made to go for a free open source alternative, either for the operating system or the application, which challenges must one expect to encounter? How can the project be managed to have the best chances of success?

Project developers must take into account that Open Source Components (OSCs) are to be used and construct the project plan accordingly. Sufficient time must be allocated for selection, evaluation, and verification of OSC.

In his article “Mission-critical development with open source software: Lessons learned,” Jeff Norris at JPL[2] describes the most critical factors to consider when designing with Open Source Software (OSS): functionality, maturity and longevity, and verification.

Functionality

The search for suitable open source components can be challenging. Projects where there is a good dialogue between the end users and the development team have the best chances of success. In many cases, an open source component has functionality close to what the end customer wants, but it is not an exact match. Either the requirements can be changed slightly or the open source community can be utilized to implement the missing features. If an acceptable match cannot be found, the alternative is to revert to in-house design for the specific component.

Projects using OSCs are particularly well suited for agile project management, where the project is run in small iterations, adding open source components one by one to build the required functionality. Each iteration starts with a brief analysis of the alternative components. The analysis results in selection of one component that is used to implement a few of the most critical use cases. If testing reveals that the resulting functionality does not meet the project requirements, the next iteration is started with another component. Once selected and properly evaluated, the new functionality can be presented to the customer for approval, and the process is repeated for the next component.

Maturity and longevity

The quality of free open source components is a concern and is often used as an argument to utilize a commercial alternative. The activity in user forums and mailing lists gives a good indication. How well established is the open source software project? Consider how much time the development team has invested, release history, community activity, bug database, functionality backlog, and so on. Does the project have a test plan? What is the quality of the test plan? How responsive is the development team to feature requests and issue reports? Some open source communities, like SourceForge.net, assign a project ranking, which is a good indicator of the overall project activity. Look for a large and diverse development team. Watch out for single developers or small project communities. These will represent a risk when their priorities change.

Verification

Incorporate the open source component into the project’s overall test strategy. Open source components often include detailed unit test and component-level tests far exceeding those common in commercial alternatives. These tests – combined with large, active user groups constantly utilizing, testing, and verifying the code – result in overall increased system reliability. Most in-house designs do not have external review, and many do not even have internal reviews.

Open source developers know that their code will be scrutinized by a number of other community contributors. One of the main motivators for contributing is to get recognition for good and well-written code. In most cases, the result is more readable and better-documented code than what can be found in commercial alternatives. An open source component selection checklist is imperative to the process (Table 1).

Table 1: Open source component selection checklist

(Click graphic to zoom by 1.9x)

Application example: Ultra-small rugged NAS

A real-world example of the process described is Galleon Embedded Computing’s recently announced Ultra Small Rugged NAS product. The requirements were nearly met by several open source NAS alternatives. An Intel hardware platform was chosen since most open source projects are developed for Intel Architecture. A Web search revealed a number of open source projects that were close to meeting the requirements: FreeNAS (BSD based), CryptoNAS and NASLite-M2 (Linux, commercial based), Turnkey Linux NAS, Gluster and OpenFiler (Linux, community based), to name a few. After a brief survey of the most promising alternatives, a more thorough evaluation was done of the two candidates that most closely met the requirements specification: FreeNAS and OpenFiler.

The two candidates are based on FreeBSD and Linux, respectively, and both alternatives met the specification with respect to functionality. SourceForge.net front-page statistics for the two projects as of December 1, 2010 indicate that FreeNAS has a much bigger user group with almost 30,000 weekly downloads totaling more than 1 million downloads in 2010. This compares to 300,000 downloads for OpenFiler, which is still sufficient to satisfy the criteria of a large and diverse user community.

Reviewing the OpenFiler backlog, it became evident that few of the issues had been addressed recently, with the latest release based on a Linux core several generations old. The latest update was almost five months old at the time of review. The FreeNAS project, on the other hand, had frequent updates, a very active user forum, and good responsiveness to a few test inquiries. The many contributors have a diverse background, and the project has a good ranking with 95 percent of the 700-plus users providing “thumbs up” feedback. A review of the negative feedback did not reveal any issues considered critical to the project.

Using an agile project process based on the aforementioned analysis, developers started the project by using FreeNAS in the first three iterations. These iterations focused on testing the file-system performance on solid-state disks, testing the network performance through four 1 Gb and two 10 GbE connections, and testing the functionality of the Web GUI. All test results showed that FreeNAS complied with the project requirements. Even so, to ensure the best possible performance of the final product, two more iterations were conducted to test the file system and network performance of OpenFiler for comparison. The recommended component selection process is depicted in Figure 1.

Figure 1: Component selection process diagram showing the component survey input

(Click graphic to zoom)

After reviewing the results of the performance tests for both alternatives, FreeNAS was chosen as the optimal solution for this project. Building the required functionality using in-house designed software would have taken years to complete, and would never have reached the same level of robustness and stability as the selected solution. Using FreeNAS, in less than three months the basic NAS functionality was ready, and the design team could start implementing specialized features not available in the standard distribution, such as GPS timeserver synchronization.

References:

www.vdcresearch.com
Jeff Norris, “Mission-critical development with open source software: Lessons learned,” IEEE Software, January 2004

Espen Bøch is VP Sales & Marketing at Galleon Embedded Computing. He has more than 15 years of experience including design and project and program management at VMETRO and Curtiss-Wright Controls. He can be contacted at [email protected].

Galleon Embedded Computing 0047 4777 3133 www.galleonembedded.com