Multi-robot coordination improved by semantic communication framework
DuoMind: Enabling Distributed Multi-Robot Coordination with Semantic Communication
RoboticsArtificial Intelligence
Summary
Coordinating multiple robots to work together on complex tasks is difficult because they need to plan carefully and act precisely. The authors introduce DuoMind, a system where each robot thinks about the big picture using language and vision models and manages actions and messages with other robots. This approach helps the robots coordinate better over long tasks. The authors also created RoboPoly, a set of tests to measure how well robots work together on these tasks. Their experiments show that DuoMind helps robots complete multi-robot jobs more effectively.
What this means in practice
- •For robotics engineers: Coordinate multiple robots in manufacturing and logistics to complete complex tasks requiring communication and long-term planning.
- •For autonomous vehicle developers: Enable fleets of self-driving vehicles to share semantic information and coordinate actions in real time for improved traffic management.
Authors
Hanchu Zhou, Dechen Gao, Hang Wang, Brendan Lynch, Boqi Zhao, Qiyao Ma, Raman Goyal, Junshan Zhang
Abstract
Vision-language models (VLMs) and vision-language-action models (VLAs) have recently driven rapid progress in general-purpose robots, yet most progress has focused on single-robot settings. Extending these capabilities to multi-robot systems remains challenging because robots must coordinate long-horizon behaviors while maintaining reliable, fine-grained execution. We introduce DuoMind, a distributed hierarchical framework for multi-robot coordination through semantic communication. Each robot uses a VLA-based action model for low-level execution and a VLM-based orchestrator for high-level reasoning and inter-agent coordination. At each planning step, the orchestrator at each robot reasons over the task instruction, local observations, and messages received from other robots. It then generates low-level instructions for the action model and semantic messages for peer robots. This architecture exploits the complementary strengths of pretrained models by combining the semantic reasoning capabilities of VLMs with the precise action-generation capabilities of VLAs. To address the scarcity of benchmarks for multi-robot coordination, we further develop RoboPoly, a benchmark comprising long-horizon manipulation tasks that require coordinated, closed-loop execution under distributed control. Experiments on RoboPoly and RoboTwin demonstrate that DuoMind improves multi-robot task performance, while ablation studies confirm the contributions of hierarchical orchestration and semantic communication. More details are available on our project page.