VGGT model knows when its camera pose predictions fail
Does VGGT Know What It Does Not Know?
Computer Vision and Pattern RecognitionMachine Learning
Summary
Sometimes, computer models that estimate 3D views from images give wrong answers but act like they are sure. The authors show that the VGGT model actually has hidden clues about when it makes these mistakes, even though it doesn't say so directly. By looking inside the model's internal data, they can predict failures much better than the model’s usual confidence scores. This insight can help systems avoid relying on incorrect camera poses by letting them know when to trust the model or not.
What this means in practice
- •For computer vision engineers: Improve 3D reconstruction pipelines by integrating failure detection to abstain from using low-confidence camera poses.
- •For robotics developers: Enhance robotic navigation systems by identifying and ignoring unreliable pose predictions from feed-forward models like VGGT.
Authors
Adityan Thirumani Srinivasan, David Nordström
Abstract
Feed-forward 3D reconstruction models such as VGGT regress cameras and geometry in a single forward pass, but they report no uncertainty for the camera poses they predict: an incorrect relative pose looks no different from a correct one. We ask whether the frozen model nonetheless knows when its pose is wrong. On HardMatch, a benchmark of 1,000 deliberately difficult image pairs on which VGGT fails on 35% of pairs, a linear probe on VGGT's frozen features predicts pose failure on held-out categories with ROC-AUC 93.5%, compared with 79.7% for VGGT's own point-map confidence and 77.7% for a classifier trained on all of VGGT's outputs. The probe catches failures VGGT is confident about, lets a downstream system abstain on the pairs it distrusts, and transfers without retraining to rotation failures on PhotoTourism and ScanNet-1500, beyond what co-visibility explains. This knowledge is absent from the per-image encoder: it is computed partway through the layers that fuse the two views, and the same holds for a second model, $π^3$. In essence, we find that VGGT knows more about its own failures than it reports.