Cross-Spectral Stereo Inertial Odometry
2026-06-29 • Robotics
Robotics
AI summaryⓘ
The authors point out that regular stereo visual-inertial odometry (VIO) systems usually rely on sensors working within the same light spectrum, which can fail together in tough environments. To fix this, they use sensors that see in different spectra, like visible light and thermal, but matching these images is hard and slow if done with current deep learning methods. Their system separates the slow matching process from the fast state estimation, allowing real-time performance. They also adjust how much they trust each sensor depending on lighting or thermal noise and handle thermal sensor quirks smoothly. Tests show their system works better in normal and difficult lighting conditions.
stereo visual-inertial odometryspectral redundancycross-spectral systemvisual-thermal sensorsdeep learning matchingreal-time state estimationphotometric entropythermal noisethermal Non-uniformity Correction (NUC)sensor fusion
Authors
Seungsang Yun, Hyunsoo Jang, Tai Hyoung Rhee, Hyunho Song, Hyeonjae Gil, Ayoung Kim
Abstract
Standard stereo VIO focuses exclusively on the benefit of metric scale via single-spectrum baselines, often overlooking the risks of spectral redundancy. This structural limitation leads to correlated failures, where both sensors simultaneously fail in degraded environments that affect their shared spectrum. Leveraging a cross-spectral system presents a complementary solution to this issue, yet the significant appearance gap between modalities renders standard matching ineffective. Existing deep learning-based matchers, while effective, introduce inference latencies that violate real-time constraints. To bridge this gap, we present an asynchronous real-time cross-spectral visual-thermal-inertial (VTI) system that temporally decouples high-latency deep matching from high-rate state estimation. Our architecture incorporates a spectral-aware weighting scheme that dynamically balances modality reliance based on photometric entropy and thermal noise, ensuring robustness against both abrupt lighting changes and thermal artifacts. Furthermore, we introduce a seamless handling mechanism for thermal Non-uniformity Correction (NUC) to maintain tracking continuity. Extensive experiments across diverse scenarios confirm that our system overcomes spectral redundancy, yielding superior accuracy in nominal daylight while ensuring robustness in visually degraded environments. We will open source our code and data: https://github.com/seungsang07/cross-spectral-stereo-inertial-odometry