Multi-robot world state model boosts task success across real and simulated robots
World Dreamer v5-lite: A Multi-Robot World Action Model with One Dreamer
Robotics
Summary
Controlling many robots at once is tricky because each one sees different things and acts separately. The authors created a method that merges all robot views into one shared understanding of the world, allowing each robot to plan its own actions while sharing useful information. They trained this model on thousands of robotic tasks both simulated and real, helping multi-robot teams perform better on complex tasks. Their model improved success rates significantly compared to previous methods.
What this means in practice
- •For robotics engineering teams: Coordinate multiple robots in shared workspaces by predicting their actions from combined camera views within a unified world model.
- •For industrial automation operators: Improve multi-robot task completion rates in manufacturing or warehouse settings by integrating sensor data and assigning robot-specific actions.$Commercial implications: Enables selling coordinated multi-robot control systems that increase efficiency and success in automated industrial environments.
Authors
Guoning Wu, Zijian Cai, Yuhang Zhang, Ying Liu, Yongbin Zheng
Abstract
Multi-robot control requires integrating distributed camera views while predicting separate action sequences. Most vision-language-action and world-action models target one robot. Combining multiple views alone does not establish spatial correspondence between them. World Dreamer v5-lite is a multi-robot world-action model built around a shared world state for robot-specific action prediction. Calibrated features from a pretrained video encoder populate this state in workspace coordinates; its spatial size remains fixed as views are added. A vision-language model provides scene context and one query per robot. The queries read the shared state and exchange pose-aware information, while their action streams remain separate. Masked-view cooperative training withholds selected camera views from both visual branches while retaining all action targets. This encourages the use of observations shared by other robots. Pretraining uses more than 10,000 hours of trajectories from 20 simulated robot types, RoboFactory, and real robots. On RoboFactory, World Dreamer v5-lite achieves 74.0% mean success, 23.7 percentage points above the strongest evaluated baseline. Across four real-robot tasks, it reaches 78.8% mean success, a margin of 13.8 points. It also reaches 58.08% overall on RoboCasa365.