Autonomous vehicles and robots are about to get even smarter

Story

April 27, 2020

Ultrafast image sensors with built-in neural networks that can sense and process optical images without latency – an application that will no doubt be useful in unmanned systems, drones, and robots in the military arena.

A group of researchers at Vienna University of Technology (TU Wien, Vienna, Austria) report that they have created neural hardware capable of image recognition within nanoseconds – and it can be trained to recognize certain objects.

Automatic image recognition is already widely used today: You may have heard about computer programs reliably diagnosing skin cancer, navigating self-driving cars, or controlling robots. Until now, these capabilities have been based on the evaluation of image data as delivered by normal cameras, which is time-consuming. When the number of images recorded per second is high, the large volume of data generated is difficult to handle.

But the TU Wien researchers opted for a different approach: By using a special two-dimensional material, they built an image sensor that can be trained to recognize certain objects. A chip represents an artificial neural network capable of learning. Because the chip itself provides ultrafast information about what it is currently seeing – within nanoseconds – data doesn’t have to be read out and processed by a computer.

Neural networks are artificial systems that operate in a similar manner to our brains. Nerve cells are connected to many other nerve cells; when one cell is active it can influence the activity of neighboring nerve cells. Artificial learning on computers works according to the same principle: A network of neurons is simulated digitally, and the strength with which one node of this network influences the other is changed until the network shows the desired behavior.

“Typically, the image data is first read out pixel by pixel and then processed on the computer,” says Thomas Mueller, a TU Wien associate professor with a research group that specializes in nanoscale optoelectronics, leading this work. “We, on the other hand, integrate the neural network with its artificial intelligence directly into the hardware of the image sensor. This makes object recognition many orders of magnitude faster.”

The researchers developed and manufactured the chip right at TU Wien. It’s based on photodetectors made of tungsten diselenide – an ultrathin material that consists of only three atomic layers. Individual photodetectors, the “pixels” of the camera system, are connected to a small number of output elements that provide the result of object recognition.

One of the chip’s features is learning through variable sensitivity. “In our chip, we can specifically adjust the sensitivity of each individual detector element. In other words, we can control the way a signal picked up by a particular detector affects the output signal,” says Lukas Mennel, who works in Mueller’s research group. “We simply adjust a local electric field directly at the photodetector. This adaptation is done externally, with the help of a computer program. You can, for example, use the sensor to record different letters and change the sensitivities of the individual pixels step by step until a certain letter always leads exactly to a corresponding signal. This is how the neural network in the chip is configured – making some connections in the network stronger and others weaker.” (Figure 1.)

Figure 1 | A picture is analyzed by the neural network chip, which then provides the appropriate output signal. Photo: Joanna Symonowicz, TU Wien.

Once this learning process is complete, the computer is no longer needed. The neural network can work alone now; if a certain letter is presented to the sensor, it generates the trained output signal within 50 nanoseconds.

The group’s goal: Fast object detection: “Our chip test is still small at the moment, but you can easily scale up the technology depending on the tasks you want to solve,” says Mueller. “In principle, the chip could also be trained to distinguish apples from bananas, but we see its use more in scientific experiments or other specialized applications.”

The technology can be usefully applied whenever extremely high speed is required. “From fracture mechanics to particle detection, in many research areas, short events are investigated,” Mueller adds. “Often it’s not necessary to keep all of the data about this event, but rather to answer a very specific question: Does a crack propagate from left to right? Which of several possible particles has just passed by? This is exactly what our technology is good for.”