World models learn from real experiences to improve robot manipulation
Direct Experience World-Model Optimization: Learning the World Beyond Action Imitation
Artificial IntelligenceRobotics
Summary
When robots try to handle delicate tasks, small mistakes can build up and cause bigger problems. The authors propose a way for robots to learn not just by copying actions but also by watching what happens during tasks, including successes and failures. This helps the robot better understand its environment and improve its actions over time, even after being deployed. They tested their method on both simulated and real robot tasks and saw better success rates.
What this means in practice
- •For robotics engineers: Improve robot dexterous manipulation by refining world models through online visual learning from both successes and failures during task execution.
- •For manufacturing automation teams: Deploy robots capable of self-improving control policies in complex, high-precision tasks by learning from their operational experiences without manual retraining.
Authors
Xiangcheng Zhan, Zirui Chen, Yicheng Zhao, Ziteng Gao, Shuo Yang
Abstract
World-Action Models (WAMs) couple action generation with predictions of how physical interactions unfold. However, current post-deployment learning paradigms typically improve behavior without requiring better world predictions. Especially in dexterous manipulation, small execution errors can compound in high-dimensional action spaces, hindering policy improvement and pushing interactions beyond the world model's training distribution. Motivated by this, we propose Direct Experience World-Model Optimization (DEWO), a post-deployment learning paradigm for WAMs that, alongside action imitation, refines world representations through visual experience to better condition action generation. Specifically, it identifies interaction turning points and learns from successful and failed futures to support classifier-free guidance. An additional value head estimates task progress from video representations and activates guidance when progress stalls during inference. Across five DexJoCo tasks, DEWO improves average success across all three WAM formulations. Ablations show that visual supervision from successful and failed continuations improves both prediction and control beyond action supervision alone. On four real-world tasks across Wuji and Sharpa, 3 x 3 grid evaluations show that two rounds of deployment learning increase success from 51.0% to 71.7% in cells with at least one initial success, a gain of 20.7 percentage points. These findings support continued predictive learning for improving control through deployment experience, making world modeling an active part of WAM adaptation.