Geometry improves sparse view synthesis with diffusion model guidance
VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Creating realistic new views of a scene from few pictures is hard. Existing methods either keep the shapes seen correctly but struggle to guess unseen parts, or generate images well but lose track of exact shapes. The authors combine geometry information with a diffusion model, which helps guide the generation using 3D points and confidence scores. This results in more accurate and stable images when imagining new viewpoints, even from few original photos. Their method works well on tests that include both views between and beyond the original pictures.
What this means in practice
- •For augmented reality developers: Generate realistic new viewpoints of scenes from limited camera angles to improve AR experiences with fewer input images.
- •For 3d content creators: Create consistent, high-quality novel views of objects or environments using fewer photos, reducing shooting and modeling effort.
Authors
Kangjie Chen, Xiangyu Li, Dongbin Zhang, Chaoda Zheng, Shijia Chen, Jinhao Deng, Hongbin Lin, Choo Sin Wai, Minqi Wang, Minghao Yang, Dake Zhong, Guorui Song, Yu Zhang, Xianming Liu, Boyang Wang
Abstract
We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.