Today's armored-vehicle and helicopter operators can detect, assess, and act upon threats with the aid of devices such as helmet-mounted displays and heads-up displays. Thanks to image processing, ground-attack vehicle commanders soon will be able to enjoy a 360-degree "window" on their environments, while pilots will obtain true spherical situational awareness around their airplanes.

Today’s armored-vehicle and helicopter operators can detect, assess, and act upon threats with the aid of devices such as helmet-mounted displays and heads-up displays. Thanks to image processing, ground-attack vehicle commanders soon will be able to enjoy a 360-degree “window” on their environments, while pilots will obtain true spherical situational awareness around their airplanes.

Enhanced and augmented vision systems are becoming faster, smaller, and more capable, and the same is true for target-detection and tracking systems. Next in the pipeline will be automatic target recognition and artificial intelligence (AI)-assisted machine vision and robotics.

Where do we go from here?

Real-time computer vision algorithms are running faster on the latest generation of massively parallel embedded graphics processing units (GPUs), which enable the silicon to do more. Larger, more complex GPUs simultaneously run more math operations on larger volumes of sensor data, while traditional processors – making their own advances – weigh more outputs, apply more complex logic, and make more decisions. Future progress in the image-processing realm will also depend on new programming tools that will enable developers to write more complex and varied applications more efficiently.

Many operations are required to translate the inputs from complex, distinct, multispectral, and distorting sensors into displays that are intelligible to humans. Among these tasks are image stitching, rotation, and fusion. In traditional programming paradigms, however, each of these functions requires hundreds, maybe thousands, of lines of code to realize.

Image stitching, for example, is a technique that enables the creation of panoramic or spherical zones of situational awareness by weaving together the inputs from multiple sensors. Image fusion, a closely related processing function, can help operators when their vision is obscured by smoke or fog. An application that dynamically selects, combines, and displays the highest-resolution inputs from visual-spectrum and infrared cameras – along with lidar [light detection and ranging] devices, for example – can assure situational awareness in degraded environments. Just imagine how much code is typically involved to perform these tasks.

The rest of the story

The other half of the image-processing success story concerns the emergence of advanced software-development tools. These resources will make it easier for military users to develop more complex and various image-processing applications without large investments in graphics-programming expertise.

The key is middleware, which is the software that mediates between the operating system, the applications, and the hardware. Middleware often features libraries and tools that help application developers get the most from the underlying hardware. Moreover, middleware can also multiply the productivity of application developers by allowing them to generate complex, high-line-of-code functions – such as image fusion, stabilization, and distortion correction – more efficiently than was previously possible.

API libraries pay dividends

OpenGL is a language that is popular among groups, such as computer game developers, who use it for programming GPUs. Although the language was invented to simplify interactions with these chips, it can be difficult, cumbersome, and time-consuming for occasional users to learn. This complexity has led to the invention of lightweight abstraction layers sitting on top of prepackaged OpenGL routines. Using these, blocks of complex OpenGL code can be integrated into a program via simple application programming interface (API) calls, streamlining the task of developing image-processing, visualization, and graphics applications.

These API libraries make programming chores easier and help programmers to be more efficient by enabling them to add functions to their applications with one or two as opposed to hundreds of lines of code, vastly reducing code-generation time and cost. In addition, the abstraction layers that are written in more accessible computer dialects such as C, can further simplify the programming task. (Figure 1.)

Easing the task of writing image-processing code means that capabilities can be created and deployed more rapidly and at a lower cost. Such tools also open the door to the development of next-generation applications such as self-driving vehicles.

Figure 1: ImageFlex provides rapid application development for Abaco’s GVC1000 small-form-factor graphics and vision system.