Depth and diffusion improve 3D reconstruction from sparse images
D3GS: Depth, DINO, and RGB Diffusion Co-Guided 3D Gaussian Splatting for Sparse-View Reconstruction
Computer Vision and Pattern Recognition
Summary
Creating 3D views from a small number of pictures is hard because it's tough to guess the shape and details of objects from limited information. The authors present a method called D3GS that uses depth maps, a special feature extractor called DINO, and generative diffusion models to better guess the 3D shapes and textures. Their approach helps improve consistency between different views and adds fine details to the 3D model. Tests show it works better than earlier methods for making 3D scenes from few images.
What this means in practice
- •For 3d graphics developers: Generate more accurate 3D models and novel views from a few photos using enhanced depth and diffusion guidance.
- •For augmented reality engineers: Create stable and detailed 3D scene reconstructions from limited camera inputs for improved AR experiences.
Authors
Yunqi Gao, Zhanfeng Liao, Hanzhang Tu, Zhaoqi Su, Guoqing Zheng, Songtao Wang, Hongwen Zhang, Zhou Xue, Leyuan Liu, Yebin Liu
Abstract
Novel view synthesis from sparse inputs remains challenging for 3D Gaussian Splatting (3DGS) due to ambiguous geometry, cross-view inconsistency, and missing details in under-constrained regions, resulting in degraded reconstruction and unstable rendering. To tackle these issues, we propose D$^{3}$GS, a Depth-DINO-Diffusion guided sparse-view Gaussian reconstruction framework that jointly enhances geometry and appearance. D$^{3}$GS first recovers a high-resolution, metric depth map via diffusion-based completion and DPT (Dense Prediction Transformer) refinement, providing robust Gaussian initialization and geometric constraints. Then, a DINO-guided view-consistent learning is introduced to augment Gaussian attributes with structural features, improving multi-view consistency. Finally, a diffusion-based Gaussian refinement module injects generative priors into an iterative optimization strategy, enhancing high-frequency geometric and appearance details within the Gaussian representation. Experiments on DTU, LLFF, and Mip-NeRF 360 show that D$^{3}$GS achieves consistent and substantial improvements over strong baselines, with ablation studies validating the effectiveness and complementary roles of each component.