Action representation pretraining improves robot hand manipulation success
Scaling Dexterous Manipulation via Action Representation Pretraining
Robotics
Summary
Dexterous robot hand movements are hard to teach because there aren’t enough robot examples to learn from. The authors separate the learning process into two steps: first, they teach the robot how to understand human hand actions by training on many human motion examples, then they use this knowledge to help robots learn tasks with fewer examples. This new method leads to much better success in complex robot hand tasks like using two hands together.
What this means in practice
- •For robotics engineers: Improve robot hand control for complex two-handed tasks by using pretrained human action data to boost learning efficiency.
- •For animation software developers: Generate more realistic hand movements by modeling task-specific action distributions from human motion data using pretrained encoders.
Authors
Yanjinghao Xu, Jiexin Zhang, Huan Zhao, Jianjia Mo, Zhenghao Yin, Dingkang Liang, Xiang Bai, Han Ding
Abstract
Large-scale vision, language, and video pretraining has substantially improved the perception and generalization capabilities of robotic manipulation models, but action generation for complex dexterous manipulation remains constrained by the scarcity of robot demonstrations. Limited robot demonstrations must be used to learn both action representations and task-conditioned action distributions, making it difficult to adequately learn action representations and, in turn, limiting the ability of downstream generative models to generate complex dexterous actions. To address this limitation, we propose ARPG (Action Representation Pretraining and Generation), which explicitly separates action representation learning from action generation: we first pretrain an action representation encoder on large-scale human manipulation data. We then freeze the encoder and train a compact generative backbone (130M parameters) to learn task-conditioned action distributions in the encoder's representation space. Experimental results show that ARPG achieves state-of-the-art performance on the DexJoCo benchmark and improves the average success rate on bimanual dexterous manipulation from 10.1% for $π_{0.5}$ to 44.7%. Scaling experiments further show that broader coverage of human manipulation behaviors effectively improves downstream performance. These results indicate that action representation pretraining provides an independent scaling dimension for improving complex dexterous manipulation capabilities.