UniWAM integrates vision language and action for better robot tasks
UniWAM: Unified World-Action Model
Robotics
Summary
People want robots and AI to understand the world, see it, and act in it. This is hard because models trained only on actions miss out on how the world works, and those trained on videos may not understand language or reasoning well. The authors created UniWAM, a system that learns from big sets of videos, actions, and language descriptions together. It combines a reasoner, a visual world generator, and an action predictor to help machines better understand, plan, and act. They tested it extensively and found it performs well on many tasks, including following instructions and handling surprises.
What this means in practice
- •For robotics engineers: Improve robot action planning by jointly learning from human and robot visual and language data for more accurate and robust task completion.
- •For interactive ai developers: Develop AI systems that understand instructions and predict future visual scenarios to better assist users in complex or long-horizon tasks.
Authors
Jiayi Chen, Wenxuan Song, Jingbo Wang, Shuai Zhou, Xicheng Gong, Zehua Fan, Ziyang Zhou, Junwu E, Haodong Yan, Fuhao Li, Qize Yu, Xu Huang, Pengwei Wang, Wen Chen, Shunbo Zhou, Haoang Li
Abstract
Vision-language-action models benefit from the understanding and reasoning capabilities of pretrained vision-language models, but action-only supervision provides limited grounding in world dynamics. Conversely, world-action models inherit spatiotemporal priors from video generation models, yet remain limited in semantic understanding and reasoning under distribution shifts. We introduce UniWAM, a unified architecture that integrates a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding of the physical world, visual generation, and action prediction. To ensure the quality of the training data, we developed a rigorous data cleaning and annotation pipeline for both human egocentric data and robot data. To adapt the vision-language component to embodied tasks while preserving its inherited language capabilities, we represent low-level actions in natural language and introduce a pre-training recipe that assigns complementary supervision from visual question answering (VQA) data, human egocentric data, and robot demonstrations to the appropriate model components. During post-training, future visual noise augmentation reduces reliance on precise future predictions, while history-conditioned flow matching uses encoded action history to initialize action generation. Together, these designs significantly reduce denoising steps while maintaining performance. UniWAM achieves state-of-the-art (SOTA) performance across multiple evaluations, including in-distribution performance, robustness, generalization, instruction following, and long-horizon task execution. Furthermore, we uncover a log-linear scaling law of unified human-robot co-training, demonstrating the effectiveness of large-scale pre-training on a mixture of human and robot data.