DXPR improves place recognition by using depth images for camera and LiDAR matching
DXPR: Depth-Based Vision-LiDAR Cross-Modal Place Recognition Using Vision Foundation Models
Computer Vision and Pattern Recognition
Summary
Robots and autonomous vehicles need to recognize their location accurately using sensors like cameras and LiDAR. This paper introduces DXPR, a system that helps match camera images to LiDAR maps even when lighting, weather, or seasons change a lot. The authors convert both camera and LiDAR data into a common depth image format, letting the same artificial intelligence model understand both types of input. They also use a smart way to compare views that accounts for how much the scenes actually overlap. Tests on well-known datasets show DXPR works better than previous methods, especially in tough conditions like night or bad weather.
place recognitioncross-modal matchingLiDARdepth imagesvision foundation modelsmetric learningrobot localizationmulti-similarity loss
Authors
Yungsoo Han, Youngseok Jang, Seungwon Roh, Jeongyeon Seo, H. Jin Kim
Abstract
We present DXPR, a depth-based cross-modal place recognition (CMPR) framework that uses vision foundation models (VFMs) to match monocular camera queries against a LiDAR map without modality-specific encoders. This enables robots and autonomous vehicles to robustly localize using only cameras within pre-built LiDAR maps, even under severe seasonal, weather, and illumination changes. The key idea is to convert both camera images and LiDAR scans into a unified depth image representation so that a single VFM backbone with an aggregation head can learn modality-invariant global descriptors. To make pairwise metric learning faithful to scene geometry, we introduce a geometry-aware overlap miner: after cross-modal scale alignment of camera and LiDAR depth, we forward-warp measurements between views to compute a pixel-level overlap score. This score relabels ambiguous pairs and adaptively modulates the positive margin in a multi-similarity loss to avoid overfitting on weakly overlapping views. Extensive experiments on KITTI odometry and Boreas demonstrate strong performance and robustness across seasons, weather, and day/night. On KITTI, DXPR achieves near-perfect Recall@1 on most sequences and outperforms prior CMPR baselines. On Boreas, DXPR achieves intra-sequence performance on par with a strong single-modal baseline (DINOv2-SALAD), while showing clear improvements in the more challenging inter-sequence setting. Compared with RangeBEV, our method consistently performs better in both intra- and inter-sequence evaluations, demonstrating robustness under diverse seasonal and illumination changes.