Compact transformer learns fast motion from event cameras and imu

Tight Learned Event-Inertial Odometry (TLEIO): a Transformer-Based Architecture for End-to-End Motion from Event Streams

Computer Vision and Pattern Recognition

Summary

Measuring motion quickly and accurately is hard in fast or dark environments. Event cameras can help because they capture changes very fast and in high detail, but existing methods are complicated and slow. The authors created a system called TLEIO that uses a transformer network to understand short bursts of event camera data and combines that with motion sensor readings, making fast and accurate motion estimates without heavy calculations. It runs faster and improves accuracy compared to previous methods.

What this means in practice

  • For drone navigation teams: Provide fast and accurate drone position tracking in challenging light conditions without heavy computation.
  • For autonomous vehicle engineers: Improve vehicle motion estimation during rapid movements or low-visibility situations by tightly fusing event-based sensing and IMU data.

Authors

Matteo Cascone, David Jensen, Alessandro Pirini, Angelo Stefanini, Pietro Venè, Marta Tintoré Gazulla, Javier Hidalgo Carriò

Abstract

Event cameras provide microsecond temporal resolution, high dynamic range, and sparse asynchronous measurements, making them compelling for odometry under fast motion and challenging illumination. Yet, accurate event-based state estimation remains difficult: existing systems often use events as a complement to frames or rely on feature tracking, correspondence search, and optimization-heavy backends, limiting the efficiency promised by event sensing. We introduce Tight Learned Event-Inertial Odometry (TLEIO), a compact filter-based pipeline that learns short-window relative motion directly from voxelized event streams and tightly fuses the resulting constraints with IMU propagation. Its learned front-end, EventsFormer, predicts local displacements and per-axis uncertainty from event voxels in a single forward pass, while a stochastic-cloning EKF fuses these overlapping predictions with IMU propagation, producing metric trajectories without explicit feature tracks and pose optimization at inference time. The network is trained directly from pose displacements, without requiring event-flow labels. We evaluate TLEIO on the TartanAir/TartanEvent benchmark and on real DAVIS240c sequences. On TartanAir, TLEIO reduces average ATE from 2.26 m to 1.10 m over the IMU-only TLIO baseline, while achieving up to 48.9% lower runtime than optimization-based event odometry methods.