Sparse view 3D surface reconstruction improved with stereo guided Gaussian splatting
Sparse-GS2Mesh: 3D Gaussian Splatting Guided by Novel Stereo Views and 2DGS for Sparse View Surface Reconstruction}
Computer Vision and Pattern Recognition
Summary
Building detailed 3D surfaces from only a few pictures is difficult because there’s not enough information to see all the shapes clearly. The authors found a way to improve this by combining a fast 3D method called Gaussian splatting with stereo vision techniques, which compare pairs of images to understand depth. They start by guessing initial depths between views, then fine-tune their model to better match stereo information, helping fill in missing details. Their method works better than previous ones when images have little overlap and performs similarly when there is more overlap.
What this means in practice
- •For augmented reality developers: Create more accurate 3D models of environments from fewer images, improving AR experiences in scenes with limited viewpoints.
- •For robotics navigation teams: Enhance 3D map-building in robots using sparse camera views, aiding navigation in complex or unknown spaces.
Authors
Younghyun Noh, Minje Kim, Tae-Kyun Kim
Abstract
Surface reconstruction under sparse-view settings remains challenging due to limited geometric cues. Volume rendering methods based on signed distance functions often produce over-smoothed surfaces, while 3D Gaussian Splatting (3DGS), though time-efficient, suffers from incomplete geometry due to the lack of reliable depth supervision and the limitation of being optimized only from given input views. In this paper, we present Sparse-GS2Mesh, a stereo-aware framework for surface reconstruction from sparse views. While 3DGS and stereo matching have been leveraged for surface reconstruction under dense view settings, we extend them to operate effectively under sparse view conditions by first initializing 3DGS using epipolar depth priors to mitigate the 3DGS overfitting problem, followed by our three key components: (I) adaptive baseline selection, (II) fine-tuning with a stereo matching network, and (III) 2D/3D co-regularized fine-tuning. Given a warmed-up 3DGS initialized with epipolar depth, the adaptive baseline selection automatically determines a baseline to synthesize for each sparse view. We then fine-tune 3DGS by backpropagating depth-refining gradients from the stereo matching network, effectively specializing the 3DGS for stereo matching. The 2D/3D co-regularization further helps obtain stable reconstruction, addressing weak geometric cues in close stereo views. Sparse-GS2Mesh achieves a 15\% improvement over state-of-the-art methods in little-overlap settings and comparable results in large-overlap settings. Codes will be publicly available.