Humanoid robots learn whole-body contact manipulation from video
ReContact: Reconstruct and Retarget Human Contacts from Video for Contact-Rich Humanoid Manipulation
Robotics
Summary
Humanoid robots often struggle to use their whole body to hold and move objects, relying mostly on their hands. This paper helps robots learn from human videos how to use other body parts for holding and supporting objects, even when parts of the body are hidden from view. The method, called ReContact, creates detailed plans about where and how a robot should touch an object to keep it stable. In tests, robots using ReContact were much better at handling objects through multiple contacts than previous methods, both in simulation and on real robots.
What this means in practice
- •For robotics engineers: Implement contact plans from video data to improve robot whole-body manipulation skills in cluttered or occluded environments.
- •For industrial automation teams: Enable robots to perform complex multi-contact object handling and transfers to improve efficiency in automated assembly or packaging tasks.
Authors
Zhongkai Gu, Yang Liu, Wei Zhu, Mitsuhiro Hayashibe
Abstract
Contact transitions allow humanoid robots to redistribute object support across the whole body, enabling manipulation beyond hand-only grasps. Human videos provide a scalable source of such transitions for robots to learn from. However, body occlusion disrupts object pose tracking and corrupts retargeted motion. Moreover, accurately reproducing the object trajectory does not ensure that the robot applies stabilizing forces at the intended contacts, risking slips or drops in later phases. To address these challenges, we propose ReContact, which reconstructs and retargets human-object interactions through an object-centric contact plan. The plan specifies contact phases, contact pairs, anchors, pressing directions, and sticking or sliding modes. Contact priors stabilize object pose reconstruction through occlusion. A residual policy over a frozen whole-body tracker is rewarded for making contact at planned anchors with forces aligned to the specified directions. In simulation, on a thigh-supported regrasp task, ReContact achieves the planned two-palm bottom hold in 83.5 of episodes over three training seeds, versus 0.16% for the best adapted baseline. One policy trained on 247 references re-solved for new placements raises in-workspace tube-transfer success from 20.4% (source reference only) to 97.8%. On a real Unitree G1, policies execute seven contact-rich manipulation and two loco-manipulation tasks.