Cooperative robots improve scene understanding with shared maps
CoRef-GS: Cooperative Referring Gaussian Splatting for Multi-Agent Scene Understanding
RoboticsComputer Vision and Pattern Recognition
Summary
Robots often need to find objects and understand their surroundings using language, but it’s tricky when multiple robots combine their separate views. The authors designed a system that helps robots share and align their 3D maps so they can understand language referring to objects seen by different robots. Their method improves alignment accuracy and object identification when robots work together. They also created a new test set with two four-legged robots to evaluate their system in both simulated and real settings.
What this means in practice
- •For multi-robot system developers: Enable cooperative robots to accurately share and interpret scene maps for object localization using natural language across different viewpoints.
- •For indoor robot navigation teams: Improve object and relation grounding in fused maps to enhance collaborative navigation and task execution in indoor environments.
Authors
Zhikun Zhou, Kunyu Peng, Runyi Yang, Junhao Cai, Di Wen, Ruiping Liu, Danda Pani Paudel, Yi Zhou, Luc Van Gool, Kailun Yang
Abstract
Referring scene understanding for embodied robots requires grounding object- and relation-centric language queries from a designated viewpoint. While a local semantic Gaussian map can support such grounding within one agent's observations, cooperative settings require this ability to remain effective after independently reconstructed maps are aligned and fused. In this setting, the referred target or its contextual landmark may come from another agent's observations, while spatial relations must still be interpreted from the querying robot's viewpoint. We formulate this problem as cooperative referring Gaussian grounding over fused maps, which requires geometric alignability, instance-level semantic comparability, and view-conditioned relation reasoning. Existing language-aware Gaussian methods mainly focus on single-map querying, whereas Gaussian registration methods optimize geometric or photometric alignment without preserving language-grounding-oriented semantic compatibility. We propose CoRef-GS, a cooperative referring Gaussian splatting framework. CoRef-GS constructs local open-vocabulary instance-aware Gaussian maps, then aligns partially overlapping maps with a cross-agent alignment module by geometric and semantic consistency, and grounds queries using a view-conditioned mask relation graph. We further introduce CoQuad-Ref, a dual-quadruped benchmark spanning both real-world and simulated indoor scenes. Experiments show that, on simulated scenes, CoRef-GS reduces the rotation error from 2.58° after coarse initialization to 0.15° after refinement, and improves real-world referring mIoU over ReferSplat from 52.6% to 68.8%. The established benchmark and source code will be publicly released at https://github.com/ruojiruoli17/CoRef-GS.git.