Dexterous robot hands learn from human touch patterns without real training
Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning
Robotics
Summary
Robots need data about how hands touch and hold things to learn delicate tasks, but this information is usually missing in human recordings. The authors found that the pattern of contacts between fingers and objects matters more than exact hand movements. They created a method that refines human recordings into realistic touch and force data, then adapts these to different robot hands using a single learning algorithm. This approach lets robots perform complex hand tasks successfully, even without training on real robots.
What this means in practice
- •For robotics engineers: Create robot hand controllers that replicate human touch patterns for precise manipulation without real-world robot trial.
- •For manufacturing automation teams: Implement dexterous robotic grasping and assembly tasks using policies learned entirely from human demonstrations.$Commercial implications: Enables production robots to perform complex manipulations learned from human contacts, enhancing automation capabilities.
Authors
Zihao Yang, Chengyuan Liu, Yu Zhou, Runze Lv, Tianyu Cui, Sheng Yi, Haohua Zhu, Irvine Lu, JieQ Sun
Abstract
Learning dexterous manipulation from demonstrations is bottlenecked by data: the contact forces that determine whether a grasp succeeds are absent from every scalable source of human demonstrations. This paper builds on two observations. First, what survives the change from a human hand to a robot hand is the contact structure of a demonstration - which finger regions touch which object locations, and in what order - rather than its joint motion. Second, physical consistency need not be engineered per task: a single residual reinforcement learning (RL) policy, trained once across diverse demonstrations, can repair kinematic recordings into physically consistent, contact-annotated trajectories, and the same residual formulation restores dynamic feasibility after retargeting. These observations yield a three-stage pipeline that converts human motion-capture recordings into dexterous robot policies with no real-robot training data: physics refinement with a simulated MANO hand recovers contacts and forces, contact-anchored retargeting transfers the demonstrated contact structure through an objective independent of hand morphology, and residual policy learning adapts the result to robot actuation. The pipeline reconstructs 25,454 single-hand trajectories (success 7.3% -> 59.3%) and 25 dual-hand tasks (16.0% -> 62.4%) with one shared policy per setting, transfers one human dataset to four morphologically distinct robot hands (+62.4 pp), and executes four contact-rich bimanual tasks on physical hardware with zero real-robot training data.