Generative model recovers accurate 3D shapes and camera views from few photos
Generative Structure from Motion with Native 3D Diffusion
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Seeing an object from just a few pictures and figuring out its full 3D shape and camera angles is very tricky because some parts aren’t visible. The authors created GenSfM, which combines smart 3D shape guessing with understanding the actual photos to fill in missing parts more accurately. Their system improves the quality of new views and shapes compared to previous methods, all without needing extra camera information or pose guesses from outside tools. This helps computers better reconstruct objects from limited images.
What this means in practice
- •For augmented reality developers: Accurately create 3D models from a small set of uncalibrated photos to enable realistic object insertion in AR scenes.
- •For robotics mapping teams: Generate consistent 3D shapes and camera positions from limited viewpoints to improve robots’ understanding of unknown environments.
Authors
Haobin Duan, Binbin Huang, Mochu Xiang, Yiqun Zhao, Zibo Zhao, Yi Ma, Shenghua Gao
Abstract
Recovering an object's 3D structure and camera motion from a few uncalibrated views is a fundamental problem in computer vision. Classical structure from motion methods recover 3D geometry consistent with the input images but cannot recover unseen regions; native generation methods produce plausible 3D objects which are related but not necessarily faithful to the input images. Methods simply concatenating reconstruction and generation allow error to cascade across the two stages, instead of producing a 3D object consistent with both the distribution and the input. We propose GenSfM, which integrates image-based 3D reconstruction with a native 3D diffusion model. In particular, we use a multi-modal diffusion transformer to jointly denoise the shape latent and per-view pose latents under shared self-attention, grounding generated geometry in the observed views while using object-level priors to infer unobserved regions. A compact 3D-native UV-volume tokenization keeps the joint sequence small as views are added, and a pose-aware stage refines geometry and texture by sampling per-voxel image features at the recovered camera poses. On Toys4k and GSO, for $4$ uncalibrated views, GenSfM improves novel-view PSNR by $3.2$-$4.5$ dB and reduces median Chamfer distance by $2.8$-$3.8\times$ over a strong native-3D generation baseline, while recovering input-view poses over $96\%$ Acc@$5^\circ$ without any external pose-estimation backbone, post-hoc optimization, or known intrinsics. Project page: https://dotb1ue.github.io/GenSfm/.