Learned uncertainty improves visual inertial state estimation and calibration
MAC-I$^2$: Learned Metrics-Aware Covariance for Robust Visual-Inertial Fusion in Initialization and Calibration
Robotics
Summary
Combining camera and motion sensor data helps machines know where they are, but often the system guesses how reliable these signals are instead of measuring it accurately. The authors present a method that learns how much to trust each sensor in different situations by predicting the actual measurement errors. This approach makes the system better at starting up and calibrating itself, especially in tricky environments like low light or moving scenes. Their tests showed big improvements in success rates and accuracy compared to existing techniques.
What this means in practice
- •For robotics engineers: Improve the reliability of robots' location tracking during initialization and calibration using learned sensor uncertainty instead of fixed assumptions.
- •For autonomous vehicle developers: Increase robustness of vehicle navigation systems in poor lighting or complex environments by adapting sensor fusion based on learned noise characteristics.
Authors
Xiang Fei, Yuheng Qiu, Can Xu, Yutian Chen, Ruogu Li, Xingxing Zuo, Wenshan Wang, Sebastian Scherer
Abstract
Visual-Inertial (VI) fusion is fundamental to accurate and robust state estimation, where camera and IMU measurements are combined according to their respective uncertainties. Existing methods, however, fuse the two modalities with predefined uncertainties, regardless of how reliable each is in the local context, and thus often struggle under challenging environments involving illumination changes, dynamic objects, and textureless regions. In this paper, we present MAC-I$^2$, which achieves robust VI fusion through learned metric-aware covariance for both modalities, so that vision and IMU compete on their own merits rather than relying on predefined uncertainties. Here, metrics-aware means that each predicted covariance faithfully reflects the actual magnitude of the corresponding measurement noise. On the visual side, we propagate learned feature-matching uncertainties into pose covariances for the fusion. On the inertial side, motivated by the observation that integration error accumulates sharply at the early stage and grows slowly afterward, we design a learned IMU model with a learnable initial covariance, and propose a dedicated fine-tuning strategy on a held-out training subset to enable the metrics-aware covariance on unseen sequences. As a showcase, we build a VI initialization and calibration system, since accurate and robust initialization and calibration are the prerequisite for any reliable VI system. Experiments on EuRoC, and VBR show that MAC-I$^2$ substantially outperforms existing methods: it achieves a 99.9% initialization success rate on EuRoC, reducing gravity and velocity errors by about 60% and 42% over the strongest baseline, and maintains 80% success rate on challenging VBR sequences where baseline methods such as VINS-Mono drop below 10%.