Summary
Robots need to understand what happens when they take different actions, but current data mostly shows only successful tries and is limited in variety. This makes it hard for robots to learn from human videos or anticipate what might go wrong. The authors created a new model that learns from human hand videos and also from likely robot mistakes, helping the robot better predict both success and failure outcomes. Their system uses clever ways to compare human and robot motions and learns from realistic failures even without exact video examples. They showed their model can generate better action-based videos and help robots plan more effectively.
What this means in practice
- •For robotic system developers: Predict how robot actions will succeed or fail more accurately by learning from both human interaction videos and modeled robot failure patterns.
- •For manufacturing automation teams: Evaluate robotic task policies for robustness by simulating potential failure scenarios, enhancing reliability in assembly line robots.
Authors
Peiyan Tu, Cong Wang, Hanxin Zhu, Xin Jin, Zhibo Chen
Abstract
Action-conditioned world models offer a promising foundation for generalist robotic intelligence by predicting the consequences of actions. However, existing robotic datasets provide limited coverage of the diverse interaction dynamics required for generalizable world modeling. Large-scale egocentric human videos offer complementary interaction experience, yet the embodiment gap between human hands and robotic manipulators hinders effective cross-embodiment learning. Meanwhile, the predominance of successful demonstrations leaves failure dynamics substantially underrepresented, limiting the ability of world models to anticipate unsuccessful executions. To address these limitations, we introduce HF-WM, an action-conditioned world model that combines Human interaction learning with Failure-aware post-training. To bridge the embodiment gap, we develop a dual-branch framework that couples video generation with auxiliary pointflow prediction, using stabilized pointflow as a unified motion representation across human and robot interactions while suppressing egocentric camera motion. To improve failure modeling, we further perform Group Relative Policy Optimization (GRPO) post-training on perturbed actions derived from successful demonstrations, guided by action alignment and physical plausibility rewards, enabling the model to learn plausible failure dynamics without paired ground-truth videos. Extensive experiments demonstrate that HF-WM achieves state-of-the-art performance in action-conditioned video generation and exhibits strong utility for policy evaluation in downstream robotic tasks.