Event and rgb cameras improve object detection across speeds
RGB--Event Object Detection Across Operating Frequencies via Reliability-Supervised Routing and Asymmetric-Resolution Fusion
Computer Vision and Pattern Recognition
Summary
Detecting objects quickly is important for things like self-driving cars. The authors show how combining fast event cameras, which track brightness changes very quickly, with regular RGB cameras can improve detection rates. They introduce a method that carefully decides how much to trust each camera at every spot in an image. This helps the system work well even when events are sparse or lighting is poor. Their approach makes object detection more accurate at high decision speeds than previous methods.
What this means in practice
- •For autonomous vehicle developers: Improve high-speed object detection systems by fusing event camera data with RGB images for more accurate and timely decisions.$Commercial implications: Enables enhanced sensor fusion software for self-driving cars that improves safety through faster and more accurate object detection.
- •For robotic vision engineers: Build vision systems that adaptively weight frame and event data per image location to maintain detection accuracy across varying speeds and lighting conditions.
Authors
Peijun Shi, Jiannan Zhao, Xiujin Zhu, Zhenguo Lao, Chee-Onn Chow, Wei Ru Wong
Abstract
Autonomous vehicles must detect objects faster than a frame camera delivers images. Event cameras record brightness changes with microsecond resolution, so RGB--event detectors can decide between frames, at an operating frequency, the rate of decisions, far above the frame rate. As this rate rises, each decision receives a sparser event window, and the reliability of the events relative to the frame varies over the scene and over time. Existing detectors weight the sensors per sample and encode the frame at the reduced event resolution. Reliability-supervised routing assigns a fusion weight to every location and supervises it with targets from the losses of two single-modality classifiers used only in training, moving loss-based supervision of fusion weights from whole samples to individual locations. Asymmetric-resolution fusion lets the frame carry the decision where the events fail: it encodes the frame at native resolution and pools every feature level onto the event grid, reusing all pretrained weights. On the DSEC-Det dataset, routing halves the accuracy loss from 20 to 180 Hz, from 6.3 to 3.1 mAP, and lowers it in every seed. The resulting detector, ERA-Det, sets a new state of the art at 63.2 mAP, 3.7 mAP above FlexEvent, leads at every operating frequency, and at 180 Hz is more accurate than FlexEvent at 20 Hz. On two DAVIS346 datasets, the router lowers the frame weight at night and in low light and raises it when the events vanish. ERA-Det upgrades released FlexEvent weights in 5,000 iterations on one GPU.