GenIA improves 3D object reconstruction from limited images
GenIA: Generative Reconstruction with Test-Time Input Alignment
Computer Vision and Pattern Recognition
Summary
Reconstructing full 3D shapes from single or few pictures is hard because the models might not perfectly match what’s seen. The authors present GenIA, a system that adjusts a powerful 3D model’s guesses at test time to better fit the observed images without needing to retrain the model. They fix object position and size using geometric clues, improve appearance by focusing on visible parts and combining observations, and use a special rendering process to guide the refinement. GenIA works well on static and moving objects, improving accuracy over several recent methods.
What this means in practice
- •For 3d content creators: Generate accurate 3D models of objects from limited photos or videos for games, simulations, or design reviews.$Commercial implications: Enables production of usable 3D assets from minimal input for digital media companies and game developers, improving workflow efficiency.
- •For augmented reality developers: Create well-aligned 3D object models from real-world camera inputs to improve AR object placement and interaction.
Authors
Stefano Esposito, Naama Pearl, Polina Karpikova, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Andreas Geiger, Jonathon Luiten
Abstract
Reconstructing complete 3D object assets from monocular or sparse multi-view observations remains challenging. Generative 3D foundation models can complete object geometry beyond the observed views, but their predictions may not faithfully reproduce the observed geometry, appearance, or pose. We introduce GenIA, a framework for test-time input-aligned generation that grounds SAM3D's generative prior in geometric and photometric observations without retraining the foundation model. We improve object pose by deriving translation and scale from geometry while retaining the learned rotation prior, and align appearance through visibility-biased attention, cross-observation fusion, and differentiable rendering guidance during denoising. An optional post-denoising refinement further adapts the appearance latent, lightweight decoder adapters, and object placement to the observations. Our framework also supports externally supplied geometry; when given temporal shapes of dynamic objects, it recovers a shared, input-aligned canonical appearance and stable world-space placement. Across synthetic and real benchmarks, GenIA improves pose prediction and object reconstruction from monocular, multi-view, and dynamic inputs, outperforming recent optimization-based, per-frame image-to-3D, and video-to-4D methods. Our project page is available at https://facebookresearch.github.io/GenIA.