Humanoid robots learn whole-body tasks from human video and motion data
WB-WAM: Heterogeneous Body-Hand Pre-training for Humanoid Loco-Manipulation
Robotics
Summary
Humanoid robots need to coordinate their whole bodies and hands to perform tasks, but existing training data often lack this comprehensive motion information. The authors developed WB-WAM, a model that learns from a large amount of human video and motion capture data which includes detailed body and hand actions. By combining diverse sources of human movement data and refining the model with special training steps, they improved robot performance in simulated and real-world tasks. This approach helps robots learn complex coordinated movements more efficiently by using human demonstrations to fill in gaps where robot data is limited.
What this means in practice
- •For robotics developers: Train humanoid robots to perform coordinated whole-body manipulation and locomotion tasks efficiently using shared human motion data.
- •For robot control engineers: Use pre-trained whole-body models to reduce the need for extensive real-robot demonstrations when developing loco-manipulation controllers.
Authors
Chuan Qin, Shaoting Zhu, Siyuan Luo, Siqiao Huang, Hongyu Zhao, Hang Zhao
Abstract
Humanoid loco-manipulation demands coordinated body and hand behavior, while conventional robot pre-training data provide limited coverage of such whole-body motion. We present WB-WAM, a World Action Model that incorporates explicit whole-body action supervision into generative video pre-training. A shared physical action space integrates body, root, and dexterous hand annotations from heterogeneous sources, enabling joint video and action learning from 1880.2 hours of partially annotated video and motion data. The resulting priors are refined through PICO mid-training and adapted to robot tasks with auxiliary forward kinematics supervision. We construct WB-Datasets to support these stages with retargeted egocentric human demonstrations and robot trajectories, allowing task-aligned human motion to supplement limited robot data. Evaluations in simulation demonstrate strong whole-body task performance with 81.9% in HumanoidArena, while real-world experiments further validate WB-WAM with 84.0% mean success across five tasks. Moreover, task-aligned PICO mid-training improves downstream task performance while reducing the need for real-robot demonstrations. These results support heterogeneous whole-body pre-training and human motion transfer as a practical route to data-efficient humanoid loco-manipulation.