Friday, October 30, 2026

The missing link between event cameras and robot vision

Penn's Fast Feature Field could make ultrafast, eye-inspired cameras as practical for robots as conventional cameras.

University of Pennsylvania School of Engineering and Applied Science

A drone with an event camera — image:
A drone equipped with an event camera, which allows it to see in a manner similar to biological eyes.
view more
Credit: Sylvia Zhang, Penn Engineering

For decades, event cameras, which emulate the biological retina by continuously monitoring changes in light intensity, have promised to revolutionize how robots perceive the world by allowing them to react faster and operate more reliably than conventional cameras permit.

But interpreting event-camera data, which resembles a dizzying stream of points rather than a recognizable series of images, has traditionally required a patchwork of specialized algorithms, each built for a single task.

Now, engineers at the University of Pennsylvania have developed Fast Feature Field, or F3, a general-purpose computational tool to efficiently transform the tens of millions of signals that event cameras produce each second into information that many different kinds of robots can use.

The tool could transform robotics by allowing for the widespread use of event cameras in platforms like self-driving cars and drones for the first time, improving robots’ ability to recognize objects, estimate distances and perceive motion in many different lighting conditions.

“We expect this to be a watershed moment in the history of event cameras, showing how to make event-camera processing reliable, fast and broadly useful,” says Pratik Chaudhari, Associate Professor in Electrical and Systems Engineering (ESE) with a secondary appointment in Computer and Information Science (CIS) and senior author of a new paper in Science Robotics. “Fast Feature Field could make the robots that increasingly fill our skies and streets much safer by helping them see the world more accurately.”

Cameras Like Eyeballs

Unlike conventional cameras, which record complete images at regular intervals (i.e., 30 frames per second), event cameras continuously respond to incoming light. “Event cameras are like the rods and cones in the mammalian retina,” says Kostas Daniilidis, Ruth Yalom Stone Professor in CIS and a co-author of the paper. “They detect changes in light, and can tell you when certain regions got brighter or darker.”

But unlike biological eyes, which have additional layers of cells to interpret that information, event cameras have never had a general way to extract meaningful patterns from their own data. “The retina has many cells beyond rods and cones,” adds Richeek Das, a doctoral student in CIS and the paper’s first author. “Ganglion cells detect motion and color, for instance, while structures like amacrine cells help suppress redundant information.”

The lack of software to take the place of that additional processing has been a central hurdle to event cameras catching on beyond limited research applications. “Without the ability to remove redundant information,” says Chaudhari, “the data from an event camera is far too noisy and voluminous to use directly.”

Predicting the Future

The researchers initially tried to solve the problem by using mathematical techniques for summarizing visual information, and sought to develop the equivalent of the Fast Fourier Transform, a widely used tool in signal processing for breaking complex signals into their component parts, for event cameras.

But after months of experimentation, the team struggled to find an approach that could reliably start from the vast information rate of an event camera and end by capturing the layout of a scene and motion within it.

Eventually, the researchers realized that they could train a neural network, a form of AI that learns from data, to discover its own way of summarizing the signals. The question was how to teach it which patterns to recognize.

The answer, they found, was prediction. “To accurately anticipate future events based on past ones, the network must learn to represent both the structure of a scene and the motion taking place within it,” says Das. “By training F3 to predict future events, we could teach it to identify the underlying patterns in the data without designing a separate approach for every task.”

Skipping the Empty Space

The result was a general-purpose way to represent event-camera data. But turning that insight into software fast enough for robots to use presented another challenge. “Our initial implementation was too slow for the rapid, real-time perception that makes event cameras so appealing,” says Chaudhari.

The solution came from recognizing that most of the possible positions where an event could occur were actually empty. “Even though an event camera can produce tens of millions of signals per second, roughly 95% of the possible positions in the data contain no events at all,” says Das.

In other words, most of the camera's pixels have nothing new to report at any given moment. “We realized we could perform calculations only where events actually occur,” adds Das, “rather than processing all that empty space.”

A Common Language for Event Cameras

Earlier this year, Sony AI unveiled Ace, a ping-pong-playing robot that uses event cameras to track the speed and spin of a ball, allowing it to compete with and defeat professional players.

However, Ace's sophisticated vision system was developed specifically for table tennis, illustrating a longstanding obstacle to the wider adoption of event cameras: Engineers have traditionally had to develop specialized software for each new application, rather than relying on existing computer-vision tools.

“Just as compression formats like JPEG and MPEG made it easier to share pictures and videos across different devices,” says Daniilidis, “F3 gives event cameras a common language, transforming their signals into a format that the enormous range of software already developed for conventional cameras can be adapted to use.”

Fast, Flexible Robot Vision

To demonstrate that versatility, the researchers tested F3 on tasks ranging from recognizing objects to estimating distances and tracking motion, using data collected from cars, four-legged robots and flying drones, all in environments that varied from urban streets at night to crowded indoor spaces.

The results combined accuracy with speed: F3-based approaches achieved state-of-the-art performance on multiple robotic perception tasks. Algorithms using F3 were roughly two to five times faster than previous approaches, and F3 itself processed high-definition event-camera data 120 times per second, or 440 times per second at lower Video Graphics Array (VGA) resolution. “F3 can enable real-time event-based perception for many different kinds of robots,” says Das.

In ongoing work with Dexter Ong and Ioannis Asmanis, both doctoral students in CIS, the team is using F3 to build autonomous drones that can navigate cluttered and poorly lit environments using event cameras. “It opens up exciting possibilities for agile drones,” says Daniilidis. “F3 has become a key component of our pipeline.”

The Future of Event Cameras

Eventually, the researchers envision baking F3 directly into the circuitry of event cameras, where it could function like an efficient camera driver, automatically transforming raw signals into a format that existing computer-vision software can readily use.

“Our goal is to make using an event camera as straightforward as using a conventional one,” says Chaudhari. “We want engineers to be able to take advantage of these cameras' extraordinary capabilities without having to reinvent the software every time they want a robot to do something new.”

This study was conducted at the University of Pennsylvania School of Engineering and Applied Science. This project was funded in part by the National Science Foundation (IIS-2145164, CCF-2212519) and the Internet of Things for Agriculture Engineering Research Center (NSF Grant EEC-1941529).