Tactile guided transfer improves robot dexterous manipulation skills
TacOT: Learning Contact-Rich Dexterous Manipulation from Human Demonstrations via Tactile-Guided Optimal Transport
Robotics
Summary
It’s hard for robots to learn fine hand movements just by watching humans because the way humans and robots touch things is different. The authors found that using touch information helps robots better understand how humans manipulate objects, even if the motion looks similar but the contact is different. They created a method called TacOT that matches human and robot behaviors based on touch signals to teach robots more accurately. This method improved robot success on several real tasks by helping the robot learn from human demonstrations with more reliable touch patterns.
What this means in practice
- •For robotics engineers: Improve robot hand manipulation by training with human demonstrations that include tactile feedback for more reliable skill transfer.
- •For manufacturing automation teams: Develop robots that perform delicate assembly tasks by leveraging human touch information to guide learning from demonstrations.
Authors
Xingting Li, Yifan Han, Zijian Lin, Wei Hou, Chuqiao Lyu, Shoujie Li, Wenbo Ding
Abstract
Learning contact-rich dexterous manipulation from human demonstrations provides a scalable source of interaction data, yet transferring such skills to robots remains challenging due to unreliable human--robot correspondence. Existing human-to-robot transfer methods typically rely on visual appearance or motion similarity, which may associate similar motions with different contact states and force patterns. Tactile dynamics provide interaction-aware cues to distinguish manipulation processes with similar motions but different contact states. We introduce tactile-guided optimal transport (TacOT), a framework for human-to-robot contact-rich manipulation. TacOT leverages action--tactile dynamic time warping to identify human--robot demonstration correspondences with consistent interaction dynamics and uses these correspondences to guide soft optimal transport alignment in a shared policy representation space. This enables human demonstrations to provide contact-rich supervision for robot policy learning without requiring predefined frame-level human--robot pairing. Across four real-world dexterous manipulation tasks, TacOT improves closed-loop success rates over action-guided OT by up to 17 points on in-distribution tasks and 20 points under targeted human-to-robot out-of-distribution transfer. Further analyses show that tactile-guided correspondence selects demonstration pairs with more consistent contact dynamics and produces latent representations that better reflect interaction-state evolution. These results demonstrate that tactile dynamics provide an effective semantic signal for establishing reliable human-to-robot correspondence in contact-rich dexterous manipulation.