Event camera detector predicts ahead to reduce latency errors
Bend the Clock: Predicting Ahead to Beat Latency in Event-Based Object Detection
RoboticsComputer Vision and Pattern Recognition
Summary
Event cameras can see fast movement with very little delay, but current detection systems still take time to process images, leading to outdated information. The authors show that existing methods lose accuracy when accounting for this processing delay. They propose ChronoFuse, a new approach that predicts what objects will look like when the results are ready, not just when the camera saw them. This method uses recent information smartly without needing future data, improving detection in fast-changing scenes like driving and drone flight.
What this means in practice
- •For autonomous vehicle teams: Improve object detection accuracy during rapid driving by predicting object states ahead to compensate for processing delays.
- •For drone flight engineers: Enhance perception systems for agile drones by anticipating object positions at output time, reducing latency-related errors in fast maneuvers.
Authors
Biswadeep Sen, Benoit R. Cottereau, Nicolas Cuperlier, Terence Sim
Abstract
Event cameras promise low-latency perception for high-speed robotic systems, where even short delays can render detections stale by the time they inform downstream robotic decisions. Yet modern event detectors still require tens of milliseconds of computation before their predictions become available. Conventional evaluation ignores this delay by comparing predictions with annotations at the observation timestamp, even though the scene may have changed by the time those predictions are produced. We study this observation-availability mismatch in event-based multi-object detection and show that state-of-the-art event detectors degrade substantially when evaluated at prediction availability rather than observation time. To address this, we introduce ChronoFuse, a causal availability-time detector that predicts object states for when its output becomes available rather than for when its input was observed. ChronoFuse performs causal cross-time fusion over a multi-scale feature hierarchy, combining current representations with cached temporal features to expose short-term temporal cues without using future observations. The fusion pathway is lightweight, adding only 0.17 million parameters and 0.84 ms of mean end-to-end latency overhead. ChronoFuse recovers 71% of the accuracy lost to latency on 1Mpx driving data and 90.8% under rapid drone motion on FRED, nearly restoring zero-delay performance. Under the extreme motion of EV-Flying, ChronoFuse reaches 20.95 sAP, compared with 2.25 for the strongest standard event detector (9.3x gain). These results show that predicting ahead can be critical for robots operating in fast-changing scenes, including autonomous driving, agile flight, and robotic interception.