Robots learn dexterous touch and vision skills from human videos
Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction
Robotics
Summary
Robots need to feel and see objects well to handle them like humans do, but videos of humans don’t show how things feel. This paper shows a way for robots to guess the sense of touch by using a computer simulation that recreates the hand and object interactions seen in videos. The simulation fills in the missing touch information, which helps train robots to use their hands skillfully. The trained robots can then pick up objects and use tools in the real world without extra training.
What this means in practice
- •For robotics engineers: Train robot hands to perform complex manipulation tasks using only videos of humans, avoiding costly robot data collection.
- •For industrial automation teams: Deploy robots capable of picking and handling varied objects and tools with touch and vision for manufacturing and warehousing.$Commercial implications: Enables sale of adaptable robot manipulators that learn from human videos, reducing data collection costs and improving deployment speed.
Authors
Ruoqu Chen, Feixiang Ruan, Liu Cao, Zihao Wang, Botian Xu, Shiqin Tong, Jiajun Liu, Mingzhi Pei, Chenyu Zhang, Wanli Xing, Kaifeng Zhang, Mengdi Xu
Abstract
Human videos are an abundant source of dexterous manipulation behaviors, but they lack tactile information that is crucial for contact-rich interaction. This raises a fundamental question: can robots learn deployable visual-tactile dexterous manipulation policies from human video demonstrations without robot-side data collection? We present DEX-X, a framework for learning visual-tactile dexterous manipulation from human videos through simulation. Our key insight is that simulation can serve as a tactile completion engine. Given monocular human demonstrations, DEX-X reconstructs hand-object interactions in simulation, where physically grounded contact dynamics provide tactile supervision unavailable in the original videos. Leveraging this recovered tactile information, we train visual-tactile dexterous manipulation policies and distill them into deployable policies operating on point-cloud observations and tactile sensing. We demonstrate zero-shot sim-to-real transfer on a dexterous hand-arm platform across diverse grasping and contact-rich tool-use tasks. The teacher policy achieves 65.9% average success across six task categories in simulation, while the distilled visual-tactile policy achieves 93% success on real-world cube picking and 53% on the challenging table-cleaning task. Zero-shot generalization to unseen object geometries is also observed on object-picking tasks. Our results suggest that simulated interaction is a key bridge between human videos and deployable dexterous manipulation policies, providing the missing physical supervision needed for scalable robot skill learning from Internet-scale human video data.