Depth based vision lidar place recognition improves robot localization
DXPR: Depth-Based Vision-LiDAR Cross-Modal Place Recognition Using Vision Foundation Models
Computer Vision and Pattern Recognition
Summary
Matching robot camera images to maps made from lidar data is challenging because cameras and lidar see the world differently. The authors designed a method that turns both camera pictures and lidar scans into similar depth images. This lets the same vision model understand both, helping robots localize better even when lighting, weather, or seasons change a lot. Their experiments show this approach works well on multiple datasets, beating previous methods.
What this means in practice
- •For autonomous vehicle engineers: Use camera images alone to localize within lidar-based maps under varied weather and lighting without new sensor-specific encoders.
- •For robotics system integrators: Enable robots to robustly recognize places using monocular cameras by matching to pre-built lidar maps across seasons and illumination conditions.
Authors
Yungsoo Han, Youngseok Jang, Seungwon Roh, Jeongyeon Seo, H. Jin Kim
Abstract
We present DXPR, a depth-based cross-modal place recognition (CMPR) framework that uses vision foundation models (VFMs) to match monocular camera queries against a LiDAR map without modality-specific encoders. This enables robots and autonomous vehicles to robustly localize using only cameras within pre-built LiDAR maps, even under severe seasonal, weather, and illumination changes. The key idea is to convert both camera images and LiDAR scans into a unified depth image representation so that a single VFM backbone with an aggregation head can learn modality-invariant global descriptors. To make pairwise metric learning faithful to scene geometry, we introduce a geometry-aware overlap miner: after cross-modal scale alignment of camera and LiDAR depth, we forward-warp measurements between views to compute a pixel-level overlap score. This score relabels ambiguous pairs and adaptively modulates the positive margin in a multi-similarity loss to avoid overfitting on weakly overlapping views. Extensive experiments on KITTI odometry and Boreas demonstrate strong performance and robustness across seasons, weather, and day/night. On KITTI, DXPR achieves near-perfect Recall@1 on most sequences and outperforms prior CMPR baselines. On Boreas, DXPR achieves intra-sequence performance on par with a strong single-modal baseline (DINOv2-SALAD), while showing clear improvements in the more challenging inter-sequence setting. Compared with RangeBEV, our method consistently performs better in both intra- and inter-sequence evaluations, demonstrating robustness under diverse seasonal and illumination changes.