Goal conditioned learning enables adaptable humanlike multi finger hand skills
Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking
RoboticsComputer Vision and Pattern RecognitionGraphicsMachine Learning
Summary
Teaching robot hands to do human-like tasks is hard because they either learn to reach goals but look unnatural, or they copy human actions but can't adapt. The authors developed a new way called AdaDex that trains robots by both mimicking a short human hand movement and practicing variations of it. This makes the robot learn flexible and natural finger movements that work well even when the situation changes. Their method helps robot hands perform complex tasks like grasping, reorienting objects, or throwing and catching.
What this means in practice
- •For robotics engineers: Train multi-finger robotic hands to perform adaptable, natural dexterous tasks from limited human motion data.
- •For industrial automation teams: Develop flexible robotic grasping and object manipulation routines that adapt to varied object poses without extensive retraining.
Authors
Yuxuan Kuang, Jiashun Wang, Yixuan Sun, Yufei Ye, Sungjae Park, Umar Iqbal, Katerina Fragkiadaki, Shubham Tulsiani
Abstract
Dexterous multi-finger hands hold promise to unlock diverse skills like functional grasps, in-hand reorientation, and dynamic handover that the human-designed world assumes, yet existing learning recipes force a trade-off. Goal-reaching reinforcement learning generalizes to arbitrary object poses but, lacking any human prior, converges to unnatural and unreliable finger strategies; motion tracking reproduces human-like behavior from a single clip but cannot deviate from it. We present AdaDex, a goal-conditioned RL framework that turns a single hand-object trajectory of a few seconds---from mocap data or HOI reconstruction from in-the-wild RGB(D) video---into an adaptable dexterous skill. At the core of AdaDex is Hybrid Dexterous Mimicking: a shared policy is trained on two populations of massively parallel environments: motion tracking environments trained with dense motion tracking objective, which instill the human prior from human motion; and generalization environments with noised initial states and goals and a sparse object-success reward, which force the skill to work far off the reference. From seconds of human motion per skill, the resulting policies can learn diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. Our method outperforms baselines in terms of generalization and functionality, and we demonstrate transfer across simulators and to the real world. Project page: https://yuxuank.com/AdaDex/.