VisTacAlign improves robot hand control using aligned human tactile and visual data
VisTacAlign: Co-Training Dexterous Policies on Tactile Human and Robot Demonstrations
Robotics
Summary
Getting robots to handle delicate tasks like assembling Lego or picking strawberries is hard because they need to sense touch and see well. This paper presents a way to teach robots using human hand demonstrations by making the robot’s visual and touch sensors see and feel like the human’s, even matching errors and perceptions. By combining these aligned human examples with robot experiences, the robot learns better and can perform precise tasks more reliably. The approach uses special sensors on a human glove and a robot hand, plus advanced AI models that understand touch and sight together.
What this means in practice
- •For robotics engineers: Improve robot hand dexterity by co-training on human and robot touch and visual data aligned to reduce sensor differences, enabling better force-controlled tasks.
- •For agriculture robotics teams: Build robots that can gently pick various-sized fruits by learning from aligned human tactile and visual sensing for precise force application.
Authors
Julien Poffet, Matthew Strong, Ankush Dhawan, Baiyu Shi, Shalika Neelaveni, Yujia Yuan, Zhenan Bao, Monroe Kennedy
Abstract
Human demonstrations are a cheap source of data for dexterous manipulation, but co-training a robot policy on them requires closing the human--robot gap in every modality the policy consumes. We present VisTacAlign, a framework for co-training 3D-visual-tactile dexterous policies on human and robot demonstrations. Glove-tracked human hand motion is retargeted to a 17-DoF tactile robot hand with a one-time fingertip correction. The human hand is then erased from both stereo views and replaced by a posed robot-hand mesh painted with pixels from robot recordings, and a real-time stereo foundation model is re-run on the composite, so the human point clouds carry the same stereo errors and visibility as the robot ones. Finally, a capacitive tactile glove is aligned to the robot's fingertip sensors in its signal space, giving one interpretable per-finger force representation. A diffusion transformer consumes point-cloud, proprioceptive, and per-finger tactile tokens. On three real-world tasks requiring precise force -- Lego assembly, plucking strawberries of varying size, and activating and lifting a power drill -- adding aligned human demonstrations to existing robot data improves over robot-only policies, and ablations show that both tactile input and visual alignment are necessary. Project page: https://vis-tac-align.github.io