DexWeave enables humanoid robots to learn dexterous hand and body movements
DexWeave: Learning Dexterous Humanoid Loco-Manipulation from Human Demonstrations
Robotics
Summary
Teaching humanoid robots to move their bodies and hands like humans is hard because their shapes and movements are different from ours. The authors created DexWeave, a system that first adjusts human motions to fit a robot's body and then teaches the robot how to perform those movements using an AI that understands body parts and hand interactions. This approach helps the robot learn better movements faster and works well on real robots doing complex tasks with their whole bodies and hands.
What this means in practice
- •For robotics development teams: Create robots that perform complex whole-body and hand manipulation tasks by learning from human demonstrations adapted to robot anatomy.
- •For robot operators: Deploy robots with enhanced dexterous object handling and locomotion for tasks requiring coordinated body and hand movements in real-world settings.
Authors
Naichuan Sun, Haotian Shen, Yizhang Zhang, Luying Feng, Haoze Wang, Yuanbo Xiangli, Yaochu Jin, Peidong Liu
Abstract
Learning dexterous humanoid loco-manipulation from human demonstrations requires transferring not only human motion, but also the coordinated interaction structure underlying the demonstrated behavior. This is challenging because embodiment differences distort the coupling among body motion, wrist placement, finger articulation, and object interaction, while kinematically accurate references may still be difficult to realize under robot dynamics. We present DexWeave, a unified framework that connects interaction-consistent motion retargeting with anatomy-aware whole-body policy learning. DexWeave first employs a two-stage retargeting procedure that initializes body and hand motions with specialized solvers and subsequently performs coupled refinement over the upper-body interaction chain while preserving lower-body support. The resulting references are tracked by an anatomy-aware Transformer policy that represents anatomical regions as structured tokens and uses directed masked attention to model their dependencies, with object information selectively conditioning the upper-body pathway for dexterous interaction. The policy jointly outputs body and dexterous-hand actions and is trained directly with reinforcement learning, without pretrained tracking policies, teacher-student distillation, or subsequent residual refinement. DexWeave improves retargeting fidelity and interaction consistency while achieving higher manipulation performance and faster policy convergence than MLP baselines. We further deploy the learned policies on a physical Unitree G1 humanoid equipped with Inspire dexterous hands, demonstrating dexterous whole-body loco-manipulation in the real world. See our project page (https://dexweave.github.io) for videos.