Robot learns from humans and its own experience to improve tasks
XPACE: Joint World and Action Modeling from Heterogeneous Experience
Robotics
Summary
Robots need to understand how their actions change the world and choose what to do next. The authors developed XPACE, a system that helps robots learn from videos showing both humans and robots performing tasks, even if the videos don’t label every action. XPACE predicts future video and the robot’s next action together, letting it imagine what happens when it moves. This lets the robot improve by practicing on its own simulated experiences and recover from mistakes, helping it handle more tasks than just copying examples.
What this means in practice
- •For robotics developers: Create robots that learn new skills from diverse human and robot video data and improve themselves by simulating recovery actions.
- •For automation engineers: Develop more robust robots that adapt to tasks with limited robot demos by augmenting training with simulated experience.
Authors
Jiacheng Wei, Jerry Bai, Xiaoyu Yue, Zidong Wang, Xiaoyang Guo, Cheng Chen, Fanqi Pu, Fan Wu, Zhixu Yue, Yizhuo Li, Feng Qiu, Bo Liu, Yuying Ge, Hui Zhou, Chenyi Chen, Yixiao Ge
Abstract
A general-purpose robot needs to draw on diverse experience, choose actions, and anticipate how those actions will change the world. We introduce XPACE, a unified embodied world model that serves as both a world action model, jointly predicting executable robot actions and future video, and a world simulator, predicting the visual consequences of prescribed actions. Our key insight is that video prediction can both connect heterogeneous experience to action learning and generate new experience for policy improvement. With a shared video backbone between the policy and simulator, we use action-unlabeled video to learn visual dynamics and action-labeled human and robot demonstrations to jointly learn video and action prediction. Building on this architecture, a coarse-to-fine training curriculum progressively emphasizes robot control while retaining human experience, allowing the policy to learn behaviors beyond those covered by robot demonstrations. Beyond learning from recorded experience, XPACE uses its simulator to create additional recovery supervision for the policy. Specifically, we adapt the simulator to its own generated context, synthesize deviation-recovery trajectories around expert demonstrations, and fine-tune the policy on filtered recovery examples. Experiments on XPENG's IRON humanoid robot show that heterogeneous training improves robustness and enables transfer of human-observed skills to tasks absent from robot demonstrations, while recovery data generated by the model's own simulator further improves real-world task completion. Together, these results demonstrate how joint world and action modeling connects learning from heterogeneous experience with simulation-driven policy self-improvement.