Geometry constrained diffusion improves 3D scene graph spatial relations
GHOST: Geometry-Constrained Hybrid Diffusion for Open-Vocabulary 3D Scene Graph Generation
Computer Vision and Pattern RecognitionMultimedia
Summary
Understanding how objects relate to each other in 3D space is hard, especially when trying to describe their positions using natural language. The authors developed a new method called GHOST that better combines geometric information with language cues to predict spatial relationships. Their approach refines object layouts using geometry-based rules and improves how relations between objects are identified. Tests show it works better than previous methods, especially for recognizing spatial connections between items in 3D scenes.
What this means in practice
- •For 3d graphics developers: Generate more accurate spatial relationship graphs for 3D scenes to improve scene understanding in graphics and simulation tools.
- •For robotics engineers: Enhance robot perception by better interpreting object relationships and positions in 3D environments using geometry-aware relation prediction.
Authors
Changqun Feng, Wangxiandi Yin, Xin Hu, Ruiting Dai, Feiyang Yang, Jielei Wang, Tao He
Abstract
3D scene graph generation requires understanding both object semantics and metric spatial relations. However, existing open-vocabulary methods often struggle with spatial predicates because language similarity reflects semantic co-occurrence rather than explicit geometry. We propose GHOST, a Geometry-constrained Hybrid diffusion framework for Open-vocabulary 3D Scene graph generaTion. GHOST jointly models continuous node geometry and discrete edge predicates, following a geometry-to-constraint-to-predicate denoising process to ensure geometrically consistent relation prediction. It introduces language-grounded constraint induction, which maps predicate embeddings to differentiable geometric constraints that guide both layout refinement and predicate reweighting. A Geometry-Topology Collaborative Denoiser further couples node and edge reasoning through noise-adaptive interaction and orientation-aware modeling. Experiments on 3DSSG demonstrate state-of-the-art performance in both closed-set and open-vocabulary settings, including over 17-point gains over RelationField on spatial predicates.