Humanoid robots learn to handle objects with whole-body control
Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions
RoboticsArtificial IntelligenceMachine Learning
Summary
Making humanoid robots interact with objects like humans do is hard because they must balance, walk, and use their hands carefully at the same time. The authors created a system called Weave that learns these skills from videos of humans manipulating objects. It transforms human movements so robots can try similar actions, controlling many robot joints together. The system works well on different objects and even on tasks it hasn't seen before. The authors also share a large dataset of robot and object motions to help others build on their work.
What this means in practice
- •For robotics engineers: Generate coordinated whole-body humanoid robot motions for interacting with various objects based on human demonstrations.
- •For animation artists: Create physically plausible humanoid-object interaction animations by using robot-executed movement data with contact information.
Authors
Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu, Mingzhi Pei, Ruoqu Chen, Mengdi Xu
Abstract
Learning humanoid-object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion. Human demonstrations provide examples of coordinated interaction, but transferring these behaviors to humanoid robots requires learning how to establish and maintain effective contacts under different embodiments and dynamics. We present Weave, a unified framework for learning whole-body dexterous humanoid-object interaction from captured human demonstrations. Weave first converts captured human-object interactions into executable robot-object references through contact-aware retargeting and approach-motion completion. At its core is a contact- and geometry-aware policy that jointly commands 29 body joints and 12 actuated finger joints across multiple objects and interaction sequences. Evaluation across nine objects yields a 92.5% success rate on trained interactions and, without any additional training, 65.0% on sequences never seen during training. We additionally release ~9,000 physically executed rollouts spanning ~23 hours, providing robot-object trajectories with contact annotations for downstream interaction-policy learning and physically consistent HOI motion generation. Project website: https://xiaohu-art.github.io/Weave/