Panographicgs improves 3d scene maps and segmentation across views

PanoGraphGS: Unifying Panoptic Segmentation and Hierarchical 3D Scene Graphs with Gaussian Splatting

Computer Vision and Pattern RecognitionRobotics

Summary

Understanding complex 3D scenes involves recognizing objects and how they fit together in a space, like rooms in a building. The authors developed PanoGraphGS, a system that uses a special 3D representation called Gaussian splatting to consistently identify objects and their relationships from different viewpoints. This helps create detailed scene maps that include both object types and their hierarchy, such as which room an object belongs to. Their system performs better than previous methods on real-world datasets and also improves 2D image segmentation.

What this means in practice

  • For robot navigation teams: Create detailed 3D maps that consistently identify objects and their room-level hierarchy for language-guided navigation tasks.
  • For augmented reality developers: Generate precise panoptic segmentations from novel viewpoints to improve scene understanding in AR environments.

Authors

Jinghan Gao, Siyu Wang, Dongyu Feng, Yiyuan Pan, Rong Xiong

Abstract

3D scene graphs support language-guided navigation by organizing physical entities and their containment and spatial relations into hierarchical maps. Building such structured maps requires a scene representation that consistently distinguishes semantic regions and object instances across viewpoints. To this end, we present PanoGraphGS, a unified framework that integrates panoptic 3D Gaussian representation learning with hierarchical 3D scene graph construction. The framework learns semantic and instance features on shared Gaussian geometry. Prototype-based semantic learning and association-based instance learning produce consistent panoptic predictions across viewpoints. The learned representation supports novel-view panoptic segmentation and direct Gaussian-level panoptic decoding. A semantic-aware scene graph module further recovers a floor-room-stuff-object hierarchy with containment and spatial relations. We further release NaviGS, a real-world dataset for panoramic scene understanding. The dataset provides 3DGS, mesh, RGB, semantic, and depth data. For 3D scene graph identification, PanoGraphGS outperforms the strongest evaluated baseline in object-node accuracy by more than 50\% points on average across Replica and NaviGS. PanoGraphGS also achieves state-of-the-art performance among the evaluated methods in 2D panoptic segmentation. These results demonstrate the effectiveness of PanoGraphGS in both hierarchical 3D scene understanding and 2D panoptic segmentation. The project website is available at: https://github.com/gaojh0333/PanoGraphGS.