Bimanual robot control learns from human hand demonstrations
DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations
RoboticsComputer Vision and Pattern Recognition
Summary
Controlling two robot hands to manipulate objects as humans do is very hard, partly because there is little data showing how humans use both hands together. The authors created DONIC, a method that learns from many human demonstrations involving two hands and many objects. They also built a large dataset called BiHOI with videos and 3D data of people using both hands together. DONIC combines learning techniques to train one robot controller able to imitate diverse bimanual tasks successfully, even with new objects it hasn't seen before.
What this means in practice
- •For robotics engineers: Develop robot controllers capable of coordinated two-hand object manipulation using human motion data to improve performance on complex tasks.
- •For automation system developers: Implement bimanual robot policies in manufacturing systems for handling diverse objects requiring dexterous two-hand coordination.
Authors
Mingze Sun, Ziang Cao, Yuan Fei, Qiankuan Tao, Zihan Wang, Tianqi Liu, Junhao Chen, Haiwen Diao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu
Abstract
Despite recent progress in dexterous reference tracking, existing models focus primarily on single-hand manipulation and a limited range of interactions, leaving the bimanual dexterous behaviors common in real life unaddressed. The scarcity of contact-rich bimanual manipulation data and the increased complexity of coordinating contacts make developing a generalist bimanual tracker particularly challenging. Therefore, we present 1) DONIC, the first generalist bimanual tracker jointly trained across operation categories and human-motion sources, together with 2) BiHOI, a multi-view dataset of 1,575 sequences spanning 229 objects and emphasizing sustained bimanual interaction, where reconstructed interactions are retargeted into robot references. DONIC combines proximal policy optimization with behavioral cloning in a feedback loop that recycles verified successful executions into training demonstrations. An object-centered per-hand reward maintains object-motion feedback while supervising each hand according to its reference contact role. Clip-Centered Advantages remove reference-dependent advantage offsets while preserving within-trajectory differences. Together, these designs enable one single policy to learn coordinated manipulation across heterogeneous references. Without finetuning, a single checkpoint achieves 86.79% success on unseen trajectories of training objects and 57.32% on held-out objects, surpassing the state-of-the-art bimanual tracking baselines by 77.36 and 45.73 percentage points.