CoGaussian improves multi-robot 3D scene understanding with open-vocabulary models
CoGaussian: Open-Vocabulary Gaussian-based Collaborative 3D Scene Understanding
Computer Vision and Pattern RecognitionRobotics
Summary
It can be hard for multiple robots working together to share what they see and understand their surroundings fully, especially when they can only recognize a limited set of objects. The authors created CoGaussian, a new way for robots to share a simple, compact 3D map made of fuzzy Gaussian shapes that include information about many kinds of objects without needing detailed labels. This method helps robots combine their views better and understand scenes more completely. Tests show CoGaussian works better than older methods that only use one robot to recognize objects in open-ended ways.
What this means in practice
- •For autonomous vehicle operators: Enable fleets of self-driving cars to share and combine 3D scene data for safer and more complete environment understanding beyond individual sensors.$Commercial implications: Allows companies to sell enhanced collaborative perception software for autonomous vehicle fleets improving safety and operational efficiency.
- •For multi-robot warehouse teams: Improve the collaborative mapping and object recognition abilities of warehouse robots working together, even with unknown or varying object categories.
Authors
Om Shivam Verma, Shaya Farahmand, Lingjun Zhao, Lu Gan
Abstract
Collaborative perception enables agents to retrieve the scene semantics and geometry beyond the field of view of any individual agent. This broader scene understanding enhances transportation safety and improves efficiency in multi-robot systems. However, existing collaborative semantic occupancy methods rely on dense 3D voxel annotations and are limited to recognizing objects within a predefined set of semantic classes. To overcome these limitations, we present CoGaussian, an open-vocabulary Gaussian-based collaborative semantic occupancy prediction framework. CoGaussian represents a scene as a set of foundation model-aligned sparse 3D Gaussians, which serve as a compact representation for communicating and fusing geometric and semantic information across multiple agents. On the Semantic-OPV2V benchmark, our method outperforms existing open-vocabulary single-agent approaches, highlighting the effectiveness of open-vocabulary collaborative scene understanding.