World-action models improve robot task success from human videos
Learning Transferable World-Action Models from Task-Paired Human Videos
Computer Vision and Pattern RecognitionRobotics
Summary
Robots can learn how to do tasks better by watching videos of humans performing similar tasks, even if the robots and humans look very different. The authors designed a system called WATCH that compares paired videos of humans doing the same task in different environments, helping the model learn what actions transfer across scenes. This training helps robots better predict future movements and actions, improving their ability to mimic human tasks. The system works well even before the robots practice on their own and continues to help afterward.
What this means in practice
- •For robot developers: Improve robot imitation learning by pre-training models on paired human videos performing the same task in different scenes.$Commercial implications: Enables more effective robot control software that better generalizes tasks from human demonstrations, boosting product performance.
- •For computer vision engineers: Enhance action and motion prediction models for varied scenarios by training on cross-video paired datasets to learn transferable interactions.
Authors
Jeongeun Park, Taekyung Kim, Dongyoon Han, Sangdoo Yun
Abstract
World-action models inherit broad visual and physical priors from video pre-training. However, standard within-video next-frame prediction allows them to rely on appearance shared between the context and future, rather than learning interactions that transfer across scenes and embodiments. We introduce WATCH, a cross-video pre-training that learns from task-paired human demonstrations collected across diverse environments. Conditioned on the video and actions of one demonstration, the model jointly predicts another demonstration's actions and future frames from its current observation. Predicting one demonstration from another encourages the model to learn task-relevant interactions that transfer across scenes and demonstrators. Our pre-training shows that these capabilities can be learned from readily scalable human-human pairs alone, without requiring matched human-robot demonstrations. Analyses before robot post-training show improved motion alignment across embodiments and better action and video prediction on unseen robots, with representations closer to those learned from robot pairs. After robot post-training, WATCH improves MimicDroid in-context success from 40.9% to 68.3% with Cosmos Policy and from 59.8% to 77.7% with Cosmos3, with the largest gains on the unseen levels. Gains persist under context-free post-training across robot embodiments. WATCH outperforms within-video pre-training and co-training baselines, supporting cross-video prediction as a scalable approach to learning transferable world-action representations.