Dexmimic improves general dexterous hand and object tracking
DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation
RoboticsArtificial Intelligence
Summary
Controlling robot hands to handle objects like humans is very hard. This paper shows a way to teach a robot hand to follow many different hand and object movements at once, so it can learn more skills without starting from scratch each time. The method uses a smart way to decide where to start learning and which parts are hardest, making training faster and better. The authors also tested their system on a real robot, showing it can work outside of simulation.
What this means in practice
- •For robotics engineers: Train robotic hands on varied manipulation tasks more efficiently by focusing learning on challenging states and phases in human-object interaction data.
- •For automation system developers: Enable dexterous robotic platforms to generalize manipulation skills to new objects or tasks without retraining from scratch.
Authors
Zhexi Luo, Zhixuan Xu, Rui Yang, Zeyu Xiao, Yiwen Hou, Lin Shao
Abstract
Achieving human-like dexterous manipulation remains a central challenge in robotics. Learning from human hand-object interaction (HOI) datasets offers a promising pathway to diverse dexterous skills, yet existing physics-based approaches train specialist policies for individual trajectories, thereby limiting skill reuse at scale. We introduce DexMimic, a generalist dexterous hand-object tracking policy jointly trained across diverse HOI trajectories. Efficient joint training requires allocating each rollout to both an undertrained phase and a starting state that supports continued tracking. DexMimic realizes this insight through Value-Guided Reset Allocation, which adapts the initialization distribution online to prioritize difficult phases and select reset states with higher expected tracking returns. We further introduce a benchmark spanning training, fine-tuning, and generalization. Experiments show improved tracking success and training efficiency, faster fine-tuning, and the potential for zero-shot generalization to unseen trajectories and objects; larger training sets further improve efficiency and generalization. Finally, we validate DexMimic on a real-world dexterous manipulation platform.