Vision based point cloud priors improve camera 3d detection and hd mapping
Leveraging Vision-Based Point Cloud Map Priors for Camera-Based 3D Object Detection and Online Vectorized HD Mapping
Computer Vision and Pattern RecognitionRobotics
Summary
Understanding 3D scenes from cameras alone is hard because depth is unclear. The authors show that if you store 3D point clouds created from past camera trips, you can use this as a map to help recognize objects and road features better in new trips. They build this map without expensive LiDAR sensors by turning prior images into point clouds enriched with semantic info. Adding this prior map to real-time camera data improves detection of objects and detailed road mapping without needing costly 3D sensors. This helps create smarter driving systems that rely only on cameras.
What this means in practice
- •For autonomous vehicle developers: Improve camera-only 3D object detection and detailed road mapping using prior point clouds derived from previous camera routes without LiDAR.
- •For robotics navigation teams: Use vision-built geometric-semantic priors as long-term scene memory to enhance scene understanding in robots without specialized 3D sensors.
Tested on one dataset.
Authors
Markus Käppeler, Rohit Mohan, Abhinav Valada
Abstract
Camera-based 3D object detection and online vectorized HD mapping provide compact scene representations for autonomous driving, but both depend on accurate metric geometry and remain limited by depth ambiguity. Over long-term deployment, observations from repeated traversals can be accumulated into persistent point cloud priors that provide geometric context beyond the current observations. Existing explicit point cloud prior approaches, however, rely on LiDAR-based map construction and therefore require expensive 3D ranging sensors. We propose a framework that constructs a static point cloud prior map from previous camera traversals using Pi3X and augments each point with DINOv3 features. At runtime, a local prior patch is retrieved using global localization, encoded with a sparse voxel backbone, and fused in bird's-eye view (BEV) with lifted multi-view camera features. Task-specific sparse transformer heads then predict 3D objects and vectorized map elements from the fused representation. On Argoverse 2, the vision-based prior improves a strong baseline from 0.287 to 0.299 CDS and from 0.669 to 0.750 vectorized mapping mAP. Ablations show that semantic DINOv3 features are particularly important for vectorized mapping. These results demonstrate that vision-built geometric-semantic priors provide an effective form of long-term scene memory for camera-based perception, improving both tasks without LiDAR for prior-map construction or online inference.