Tartanmatch improves multi-sensor image matching across all sensor types
TartanMatch: Towards Universal Dense Correspondence Across Modalities
Computer Vision and Pattern RecognitionMachine LearningRobotics
Summary
Matching points between images from different sensors like cameras, thermal sensors, or laser scanners helps robots understand their environment better. Existing methods usually only work well for certain sensor pairs and need many separate models. The authors designed TartanMatch, a single model that can find matches across 25 different pairs of sensors by learning from both real and synthetic data. This model performs much better than previous methods, especially when matching images from different sensor types. Training one model for all sensor types also improves accuracy compared to training separate models for each pair.
What this means in practice
- •For robotic system developers: Build robot vision systems that robustly match features across a wide range of sensor types using a single model.
- •For autonomous vehicle engineers: Improve sensor fusion for autonomous cars by using one model to align data from cameras, LiDAR, thermal, and other sensors.$Commercial implications: Enables production of better multi-sensor perception modules for autonomous vehicles, improving safety and environment understanding.
Authors
Hyeokjoon Kwon, Jiting Cai, Ruogu Li, Kritan Bhandari, Geethika Hemkumar, Parv Maheshwari, Yuheng Qiu, Yuchen Zhang, Sebastian Scherer, Wenshan Wang
Abstract
Robotic platforms combine multi-modal sensors to operate across diverse conditions, but most dense correspondence methods are limited to RGB-RGB or a small set of modality pairs. Training a model for each modality pair scales poorly with the number of sensors and forgoes potential positive transfer across modality pairs. We introduce TartanMatch, a unified model for dense correspondence across all 25 ordered pairs among RGB, event, thermal, depth, and projected LiDAR. TartanMatch adapts an RGB-pretrained correspondence model to all modality pairs within one shared network using lightweight modality adaptation and a mixture of real and synthetic data. On zero-shot real-world benchmarks, TartanMatch reduces Endpoint Error (EPE) over the strongest baselines by 61.9% for cross-modal matching and 49.6% for same-modal matching. We further show that joint training across all modality pairs improves average matching accuracy over pair-specific training, especially on challenging cross-modal pairs. These results demonstrate that a single model achieves broad multi-modal coverage and state-of-the-art matching accuracy.