Accelerating multi-view 3d geometry with lighter attention mechanisms
GPA-VGGT: Geometry-Preserving Acceleration of VGGT
Computer Vision and Pattern Recognition
Summary
Multi-view 3D perception systems need to process many images to understand a scene’s shape and camera positions. The VGGT method does this in one step but slows down a lot as more images are added due to complex calculations. The authors developed GPA-VGGT, which speeds up VGGT without needing extra training by simplifying these calculations early on while keeping important details. This keeps the 3D estimates almost as accurate while making the process much faster.
What this means in practice
- •For robot control engineers: Accelerate multi-camera 3D scene reconstruction pipelines to improve real-time robot navigation without accuracy loss.
- •For augmented reality developers: Speed up 3D geometry recovery from multiple views to enable smoother AR experiences on devices with limited compute.
Authors
Muyeol Sung, Jangwon Kim, Jongwoo Lim
Abstract
Feed-forward geometry models such as VGGT recover camera parameters, depth maps, and point maps from one to hundreds of views in a single forward pass, making them powerful backbones for multi-view 3D perception. However, VGGT's global attention is quadratic in the number of frames and dominates end-to-end runtime on long sequences, which has motivated a range of training-free efficiency methods such as token merging. These methods, however, come with a drop in accuracy that makes them hard to use in place of VGGT. We propose GPA-VGGT, a training-free acceleration method that replaces global attention in early blocks with frame attention augmented by a shared cross-frame K/V set. In the remaining blocks, we merge only keys and values while preserving all queries, restore the attention mass of the tokens each merged key represents, and randomly project the matching metric to 128 dimensions to reduce matching cost. Across diverse multi-view geometry tasks, GPA-VGGT provides training-free acceleration at speedups comparable to prior work while preserving the information required for precise geometric estimation, retaining over 99% of VGGT's pose accuracy under AUC@3.