Transformer model directly matches images to 3D scans for pose estimation
DRS-VPT: Directly Relocalizing in a Scan with Vision Point Transformers
Computer Vision and Pattern RecognitionRobotics
Summary
Pinpointing the exact position and angle of a camera inside a 3D scan is important for things like self-driving cars and indoor navigation. The authors created a way to use transformers—a type of AI model—to directly connect images to 3D point clouds without needing special tuning for each map. Their model not only figures out where the camera is but also aligns points between the image and the scan. It works well on driving scenes and indoor environments, even ones it hasn’t seen before.
What this means in practice
- •For autonomous vehicle engineers: Reference a single model to align vehicle cameras with LiDAR scans, improving localization without map-specific retraining.
- •For indoor navigation developers: Enable robust indoor camera-to-map relocalization across different environments using one universal model.
Authors
Lanke Frank Tarimo Fu, Maurice Fallon
Abstract
We present DRS-VPT, a feed-forward transformer architecture for foundational image-to-scan registration. Given query images and a reference 3D point cloud, the model predicts the scan pose and point map alongside the poses and point maps of each camera, all expressed in the first camera's frame. It additionally predicts a coarse-to- fine pyramid of per-point and per-pixel features for direct reprojective alignment of the scan to the first image. This formulation unifies downstream tasks such as camera-LiDAR calibration in autonomous driving and indoor camera-to-map relocalization. A single DRS-VPT model achieves state-of-the-art performance for image-to-LiDAR registration in autonomous driving, competitive indoor relocalization without training map-specific weights, and strong zero-shot transfer to unseen environments. We also show qualitatively that the model learns complex scan-to-image projection properties such as occlusion of back-facing points.