Gaussian splatting method fills 3D scenes faster without reference images
WINGS: Reference-Free Gaussian Splatting Inpainting with 3D-Native Generative Priors
Computer Vision and Pattern Recognition
Summary
Editing 3D scenes often requires filling in missing parts so they look natural. Previous methods used 2D images for this, but that could cause inconsistencies and slow processing. The authors created a new method that works directly in 3D using a pre-trained model to guess the missing shapes and appearances. This approach avoids problems with 2D images and runs faster. Their tests show it produces good results both visually and by user feedback.
What this means in practice
- •For 3d content creators: Automatically fill missing parts in 3D scenes to enable faster and more consistent scene editing without manual image-based fixes.
- •For augmented reality developers: Generate plausible 3D content to complete real-world scans with missing geometry for seamless mixed reality experiences.
Authors
Noé Lallouet, Michael Fischer, Elie Michel
Abstract
Inpainting 3D Gaussian Splatting scenes, a key challenge in 3D editing, requires generating plausible content within a masked region of 3D space. Prior approaches rely on 2D diffusion models to produce one or several inpainted reference views, making them susceptible to challenges associated with multi-view inconsistency and lengthy optimization times. Departing from these approaches, we introduce a reference-free Gaussian splatting inpainting method operating natively in 3D. Our method leverages the embedding space of a large, pre-trained 3D prior, combined with a structure completion network to feed a generative prior which reconstructs the missing region's geometry and appearance. Performing content generation entirely in 3D, it avoids the need to reconcile inconsistencies of multiple inpainted reference images, and is faster than related 2D-based methods. We demonstrate the effectiveness of our method qualitatively and quantitatively, through extensive experiments and a user study. To the best of our knowledge, this work is the first Gaussian splatting inpainting method to operate in the learned representation space of a 3D-native generative prior without relying on inpainted reference views.