LiDAR maps improve camera localization with robust visual matching
From LiDAR Maps to Visual Localization: Unified Visual Association for Robust Point-Line-Plane Pose Estimation
Robotics
Summary
Robots often need to know exactly where they are using cameras and maps made by laser scanners called LiDAR. It’s hard for robots to match what a camera sees with a LiDAR map because the two types of data look very different. The authors created a clever way to turn the LiDAR map into images that look like camera pictures so the robot can match things more easily and figure out its exact location. Their method works well even when lighting changes or things move around.
What this means in practice
- •For robotics developers: Enable robots to use existing LiDAR maps to localize themselves accurately using only cameras, even under changing lighting conditions.
- •For autonomous vehicle teams: Support precise vehicle position tracking by combining camera views with prior LiDAR maps without needing specialized cross-modal matching models.
Authors
Wentao Zhao, Zikun Chen, Yihe Niu, Haoyu Chen, Jingchuan Wang
Abstract
Camera localization in a prior LiDAR map provides a persistent geometric reference for long-term robotic navigation, yet remains challenging because of the substantial modality gap between camera images and point-cloud maps. We present a unified localization framework that makes the LiDAR map visually addressable rather than relying on a dedicated image-LiDAR correspondence model. Map geometry and reflectivity are rendered into LiDAR-derived quasi-images with explicit 2D-3D provenance, enabling camera observations and rendered map views to share mature visual features and matchers for both global localization and continuous pose tracking. Point and line correspondences are established through this common visual interface, while the retained provenance recovers metric LiDAR geometry and line-supported planar constraints for pose estimation. To improve robustness under ambiguous associations and weak geometry, we further introduce a distribution-aware, observability-complementary optimization strategy. Instead of reducing matching ambiguity to a scalar confidence, candidate association distributions are propagated into directional pose-information uncertainty, and reliable structural factors are selectively reinforced according to their ability to complement the currently weak pose directions. Experiments on the EuRoC MAV benchmark and self-collected real-world sequences demonstrate accurate global localization and robust continuous 6-DoF tracking using only a pre-built LiDAR map as the persistent prior, including under severe illumination variations and dynamic occlusions.