Spackle improves novel view synthesis from single images with adaptive gaussians
Spackle: Completing Large View Single Image NVS with Adaptive Gaussians
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Novel view synthesis is a way to create new images of a scene from different angles using just one photo. The problem is that when the new viewpoint is very different, parts of the scene that were hidden or unclear get harder to recreate accurately. The authors found that using a fixed number of simple shapes called Gaussians to model the scene limits detail and creates competition for representation space. To fix this, they designed Spackle, a method that first builds a basic scene model, spots where it looks wrong, and adds extra Gaussians only in those tricky areas. This approach helps produce better images when looking at the scene from very different angles.
What this means in practice
- •For augmented reality developers: Create more realistic and detailed augmented reality scenes when users change viewpoints dramatically using fewer computational resources.
- •For game engine teams: Improve real-time rendering quality in games by dynamically enhancing scene details in newly visible areas from novel camera angles.
Authors
Xuanzhi Liu, Yuhe Zhou, Xinyi Wu, Zhenyao Wu, Jinghao Chen, Ruize Han, Song Wang
Abstract
Single-image novel view synthesis (NVS) enables photorealistic rendering of un- observed viewpoints from a single input. Practical NVS systems require two key capabilities: robust reconstruction of occluded regions and high inference effi- ciency. While hybrid decoupled frameworks combining feedforward 3D Gaussian Splatting (3DGS) and diffusion models show promise for large-view-deviation NVS, they suffer from capacity competition: a fixed number of Gaussians forces resource shifts from visible to newly disoccluded areas, degrading original scene fidelity when the target view deviates significantly from the input. To address this, we propose Spackle, a lightweight residual learning framework that mit- igates capacity competition without sacrificing efficiency. Spackle operates in three stages: predicting base 3DGS attributes from given views, automatically identifying poorly reconstructed regions, and learning a residual 3DGS optimized exclusively for these areas. At inference, we combine the baseline and aug- mented Gaussians for NVS. We conduct comprehensive experiments and show that Spackle achieves state-of-the-art performance on large-view-deviation cases.