Pointvggt registers multiple rgb-d scans without training or pairwise steps
PointVGGT: Zero-Shot Multiview RGB-D Point Cloud Registration with Visual Geometry Foundation Priors
Computer Vision and Pattern Recognition
Summary
Aligning multiple 3D scans taken from different positions is hard because errors can build up and slow down computations. The authors present PointVGGT, which skips the usual step-by-step alignment between pairs of scans. Instead, it uses a visual geometry model to estimate the overall positions of all scans directly, then refines this using 3D spatial information efficiently. This method works well without any special training and is fast and accurate on many types of 3D scan datasets.
What this means in practice
- •For robotics developers: Automatically align 3D scans from robot-mounted RGB-D sensors without manual calibration or pairwise scan matching.
- •For augmented reality app makers: Create coherent 3D maps from multiple unordered RGB-D images in real time without training data for AR scene understanding.
Authors
Haobo Jiang, Liang Yu, Jianmin Zheng
Abstract
This paper addresses multiview RGB-D point cloud registration, aiming to estimate global rigid poses for unordered RGB-D scans and align them in a metrically consistent coordinate frame. The conventional pairwise-then-global paradigm suffers from locally optimized pairwise registration, severe error propagation and high computational burden. In particular, existing methods typically treat RGB data as a mere auxiliary matching cue and overlook the holistic geometric priors (e.g., camera poses and 3D models) encoded across image sequences. This paper introduces PointVGGT, a zero-shot framework built upon a novel \emph{foundation-then-refinement} paradigm that systematically leverages visual geometry foundation models (e.g., VGGT) as the computational backbone for robust, training-free multiview RGB-D registration. In the foundation stage, we directly recover metrically consistent global poses (without any pairwise estimation) by grounding the scale-ambiguous pose predictions of the foundation model against metric depth observations. In the refinement stage, we introduce an efficient voxelized spatial hashing mechanism that exploits the globally coherent 3D reconstruction (induced by the foundation model) as a shared spatial anchor, enabling dense multiview correspondences in near-linear time. On top of this, an IRLS-based robust motion-only bundle adjustment is performed using a conjugate gradient solver to jointly minimize the correspondence and reprojection residuals for multiview pose refinement. Extensive experiments on indoor/object-centric/outdoor datasets verify the outstanding zero-shot registration accuracy and computational efficiency of our proposed method.