Perception prior improves 3D scene segmentation and shape reconstruction
PePESeg3D: Perception Prior Enhances Multi-Scale Segmentation for 3D Gaussian Splatting
Computer Vision and Pattern Recognition
Summary
Segmenting 3D scenes into meaningful parts usually treats geometry and semantics separately, which limits accuracy. The authors present PePESeg3D, a new approach that uses perception knowledge like depth and object masks early in both shape reconstruction and feature learning. This leads to better alignment of 3D shapes with their semantic parts, improving segmentation at multiple scales. Their tests show PePESeg3D works better than previous methods on several benchmarks for both segmentation and 3D reconstruction.
What this means in practice
- •For 3d graphics developers: Improve accuracy of multi-scale 3D object segmentation and reconstruction for realistic scene modeling and editing.
- •For augmented reality engineers: Create more semantically aware 3D reconstructions that support robust object recognition in AR environments.
Authors
Sungjae Choi, Seunghee Koh, Junmo Kim
Abstract
Recent advancements in 3D Gaussian Splatting (3DGS) have extended its capabilities to multi-scale segmentation. Existing methods reconstruct a scene with Gaussian primitives and learn multi-scale segmentation features separately, which leaves the geometry unaware of semantic structure and the feature learning dependent on incomplete mask supervision. To address these limitations, we present PePESeg3D, a novel framework that injects perception priors into a multi-scale 3D Gaussian segmentation pipeline. To fully exploit perception priors, we integrate them not only into contrastive feature learning but also into the upstream geometry reconstruction. Specifically, PePE Reconstruction incorporates monocular depth and mask constraints to ensure semantically coherent object structures. Building on this aligned geometry, PePE Contrastive Learning leverages dense depth-color cues and view-consistent centroid supervision to compensate for the incompleteness of multi-scale masks obtained from a 2D foundation model. Extensive experiments on the SPIn-NeRF, LERF-Mask, and NVOS benchmarks demonstrate that PePESeg3D achieves state-of-the-art performance in both multi-scale segmentation and scene reconstruction, highlighting the importance of integrating perception priors into both geometry optimization and feature learning for accurate multi-scale 3D segmentation. Our code is available at https://github.com/BeCow5X5/PePESeg3D.