Magic-W0 model improves robot control by linking 3D world state and actions
Magic-W0: A Structured World-Action Foundation Model for Physical Intelligence
Robotics
Summary
Robots need to understand how their actions change the world to do tasks well. The authors introduce Magic-W0, a model that predicts how the 3D environment and objects move when a robot acts, while also helping decide what action to take next. It uses structured representations combining visual and language context with 3D geometry and motion, allowing the robot to plan and adapt better. Tested on simulated and real robots, Magic-W0 shows improved performance and flexible learning from fewer examples.
What this means in practice
- •For robotics engineers: Improve robot manipulation by integrating structured 3D world understanding with action planning for better task success and adaptability.
- •For simulation platform developers: Develop simulation tools that incorporate structured world-action models to generate realistic task scenarios and test robot policies.
Authors
Xuhua Chen, Zhenhan Yin, Yuan Zhang, Lingfeng Zhang, He Zheng, Tong Mu, Shun Zuo, Dian Zhou, Di Wu, Xuan Zhou, Shaojie Wan, Rongtian Shen, Qiulong Xu, Yiduo Li, Yinglong Wang, Yanqian Wang, Kun Wang, Tao Zhang
Abstract
World-action models (WAMs) augment robot policies with action-conditioned environment dynamics, yet existing approaches largely rely on future observation reconstruction or generic latent prediction and lack structured, control-oriented world representations tightly coupled with action generation. We introduce Magic-W0, a world-action foundation model that jointly models structured physical state evolution and continuous actions. Magic-W0 represents interaction as a Structured World Transition consisting of Current State, Transition, and Future State. Current State combines vision-language context with Current 3D Geometry; Transition is represented by 3D Motion capturing action-induced three-dimensional changes; and Future State is represented by Future Semantics describing task-relevant outcomes. To couple prediction and control, we propose a layer-aligned world-action interaction architecture in which evolving action hypotheses condition world-transition prediction, while predicted world representations continuously inform action generation. Magic-W0 is pre-trained on large-scale egocentric human manipulation, UMI, real-robot, and simulation data, with latent supervision for geometry, 3D motion, and future semantics from pre-trained visual models. Inference-time interventions show that structured world representations respond systematically to changes in candidate actions and that action-related information propagates through shared 3D representations into future semantic predictions. On RoboDojo-Sim, Magic-W0 achieves an average Score of 27.10, the highest among the compared WAMs. Across multiple real-robot tasks, it also demonstrates strong downstream performance after fine-tuning with limited downstream data, supporting generalization and rapid adaptation.