Vision language models enable robots to learn from human videos
ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning
RoboticsArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Teaching robots to do tasks by watching humans is hard because robots need to understand the actions precisely. The authors created a new way to help robots imagine the effects of actions by breaking them down into small steps, called chains of dynamics, which describe how the world changes. This approach lets robots learn not only from labeled robot actions but also from lots of human videos without action labels, improving how well robots can do new tasks they haven't seen before. Their tests show this method works both in computer simulations and real robots.
What this means in practice
- •For robotics engineers: Use human video data without action labels to train robots that adapt to new tasks and environments more effectively.
- •For industrial automation teams: Build robotic systems that can learn from diverse human demonstrations to perform complex manipulation in varied factory settings.$Commercial implications: Enables manufacturing robots that quickly learn new tasks from broad human video data, improving flexibility and reducing programming cost.
Authors
Chaehyun Kim, Chi-Pin Huang, Jisoo Kim, Min-Hung Chen, Yu-Chiang Frank Wang, Seungryong Kim, Fu-En Yang
Abstract
Vision-Language-Action (VLA) models have made substantial progress in robotic manipulation, but their generalization remains constrained by the diversity and coverage of action-labeled robot demonstrations. Human videos provide much broader behavioral and environmental experience, while learning from them faces a fundamental trade-off between scalable data and explicit action grounding. We introduce ThinkWorld, a VLA learning framework that grounds action reasoning in world dynamics and strengthens it through reinforcement learning with world imagination. ThinkWorld represents each action thought as a Chain-of-Dynamics (CoD), a sequence of discrete motion tokens that captures the intermediate dynamics connecting the current world to a future state and serves as a common action representation across embodiments. We first initialize CoD with action-labeled demonstrations, then predict the future induced by each CoD and reinforce those whose predictions better match the observed outcome. This enables videos without action annotations to provide direct supervision from observed state transitions, allowing human and robot experience to refine the same reasoning space. Experiments in simulation and on real robots demonstrate effective human-to-robot transfer and strong generalization to unseen tasks and environments.