ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views
Computer Vision and Pattern Recognition
Summary
The authors present ReconSplat, a new model that reconstructs 3D scenes in a way that balances generating believable new views with keeping accurate scene geometry. Their method uses 3D Gaussian splatting as a middle step and combines it with a multi-view latent diffusion model to refine and fill in details for both appearance and shape. By guiding this process with special 3D features, they ensure the results stay geometrically consistent. Tests on real-world datasets show that ReconSplat creates realistic new views and precise depth maps, especially when looking from difficult or unseen angles.
Authors
Giuseppe Stracquadanio, Kevin Raj, Julia Grabinski, Stefan Roth
Abstract
We introduce ReconSplat, a feed-forward model for 3D scene reconstruction that aims to address the longstanding trade-off between plausible view generation for unobserved regions and geometric consistency, providing both geometrically aligned novel views and sharp depth estimates. Our approach builds on 3D Gaussian splatting (3DGS) as an intermediate differentiable scene representation and integrates it with a multi-view latent diffusion model (MV-LDM) trained to act simultaneously as a refiner and an inpainter for appearance and scene geometry. We enforce geometric consistency by guiding the diffusion process with variational 3D latent features for appearance and geometry, encoded by the feed-forward 3DGS representation and rasterized to 2D latent space. ReconSplat produces both photorealistic novel views and accurate depth maps on real-world benchmarks, RealEstate10K and DL3DV-10K, outperforming existing methods in challenging extrapolation setups. Notably, ReconSplat allows the extrapolation of unseen and challenging viewpoints jointly with coherent and precise scene geometry.