Seeing the Unseen: Semantic-in-Gaussian for Sparse-View 3D Generalization
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present SeeU, a new method to improve 3D Gaussian Splatting for creating novel views from limited camera angles. Their approach uses semantic information to better fill in parts of a scene that are partially seen or blocked, which existing methods struggle with. They introduce a module that combines information from multiple views to refine 3D models in a way that keeps surfaces consistent and complete. Experiments show that SeeU works better than previous state-of-the-art methods, especially when trying to predict views far from the original cameras.
3D Gaussian Splattingnovel view synthesissemantic conditioningmulti-view aggregationPSNRsurface reconstructiontransformersocclusion handlingfeed-forward inference
Authors
Zeyang Bai, Yunpeng Wang, Yunbiao Wang, Jun Xiao
Abstract
Generalizable 3D Gaussian Splatting (G-3DGS) has emerged as a promising approach for novel view synthesis undersparse-view settings. However, existing frameworks remain restricted by pixel-aligned Gaussian estimation, whichstruggles in partially observed or occluded regions and often leads to incomplete surfaces or structural collapse. Toaddress these challenges, we propose SeeU (Seeing the Unseen), a novel G-3DGS framework. We frame its core design asSemantic-in-Gaussian: semantic-conditioned refinement in Gaussian space. Specifically, we introduce a Cross-viewEntropy-Aware (CEA) module that aggregates multi-view semantic and geometric cues into compact embeddings. Theseembeddings guide the Conditional Gaussian Transformer, which applies residual updates to coarse Gaussians, helpingrecover under-constrained regions of partially observed structures while preserving surface consistency. Comprehensiveexperiments on multiple benchmarks demonstrate that SeeU consistently improves rendering quality and structuralcompleteness while retaining efficient feed-forward inference. Especially under challenging extrapolation settings,SeeU achieves an average improvement of 2.44 dB in PSNR compared to recent SOTA G-3DGS methods.