Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
2026-08-26 • Robotics
RoboticsComputer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of teaching robots to perform new manipulation tasks they haven't seen before. They use a method called in-context learning, where the robot learns what to do by watching a human video showing the task, instead of updating its program. To train their system, they created a large dataset matching human videos with robot actions and designed a special training method to make the robot rely on the videos for guidance. Their approach, Zero-WAM, showed much better success in simulation and worked well on real robot tasks involving multiple objects and detailed actions. This helps robots better understand and follow complex tasks just by watching humans.
zero-shot generalizationrobot manipulationin-context learningvideo-action modelhuman video guidancerobot trajectorytask specificationfuture chunk predictionRoboTwin simulationmulti-object manipulation
Authors
Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, Yinghao Xu
Abstract
Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos, yielding HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks. For model training, we further introduce an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline. In real-world evaluations, it follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion.