Whole-body robot actions improved by learning from human and humanoid motions
WholeBodyWAM: Learning Whole-Body World Action Models with Scalable Motion Priors
Robotics
Summary
Robots that move their entire bodies to interact with the world need good models to predict how to act. Collecting lots of robot motion data is hard, but there’s plenty of movement data from humans and other humanoids. The authors created a way to learn from this large but varied motion data to build a general motion model. This model helps robots better predict and produce whole-body movements for tasks, even with limited robot-specific training. Their experiments show this approach improves robot motion prediction and real-world manipulation.
What this means in practice
- •For robotics engineers: Train humanoid robots to perform complex whole-body tasks using pretrained motion models from large human and humanoid datasets.
- •For robot control software teams: Improve robot manipulation control software by integrating motion priors learned from diverse humanoid actions to reduce robot-specific training data.
Authors
Bowei Zhang, Qiyao Zhang, Shuanghao Bai, Xinhua Wang, Meng Li, Yilei Wang, Leiwang Zhang, Jian Tang, Lu Zhou, Lei Sun, Zhengping Che
Abstract
Humanoid whole-body manipulation requires coordinated whole-body dynamics, yet large-scale trajectories from a target robot are expensive to collect and difficult to scale. In contrast, whole-body motion from human and humanoid sources is abundantly available, although such data cannot be directly used as embodiment-specific robot actions. This work asks whether these scalable motion resources can instead provide a transferable predictive prior for humanoid world-action modeling. We introduce WholeBodyWAM, a humanoid world-action model that learns whole-body dynamics from large-scale heterogeneous motion before target-robot training. We curate UniMotion-4K, a motion corpus spanning more than 4K hours from human videos, native 3D motion datasets, and heterogeneous humanoid platforms, and canonicalize these diverse sources into a unified motion space. A language-conditioned Motion Expert is then pretrained to predict future whole-body motion without target-robot action supervision. During robot post-training, the pretrained Motion Expert is integrated with Video and Action Experts through asymmetric Mixture-of-Transformers (MoT) attention, enabling predictive scene dynamics and whole-body motion to jointly inform embodiment-specific action generation. Experiments show that WholeBodyWAM consistently benefits from increased motion-pretraining scale, improves future-motion prediction and downstream task performance, and transfers effectively to real-world humanoid manipulation. Moreover, the pretrained motion prior substantially improves data efficiency under limited target-robot demonstrations.