Dexterous robot hands improve grasping with touch sensing models
Temporal Visuo-Tactile Learning for Dexterous Grasp Stability
RoboticsComputer Vision and Pattern RecognitionMachine Learning
Summary
Grasping objects reliably is tricky for robots, especially when they just use cameras. The authors show that giving a robot hand sensitive touch sensors on its fingers helps it better predict if it’s holding something firmly. They trained computer models using a large dataset with many grasp attempts and different objects to learn from both vision and touch over time. When the robot used these touch-based predictions to adjust its grip, it successfully picked up objects more often. This work shows that detailed touch sensing combined with smart models can help robots handle delicate tasks more like humans do.
What this means in practice
- •For robotics engineers: Use combined touch and vision models to improve robotic hand grasp reliability and enable model-guided regrasping for varied objects.
- •For prosthetics developers: Incorporate high-resolution fingertip tactile sensing with temporal models to enhance grip control and stability feedback in dexterous prosthetic hands.
Authors
Ken Nakahara, Aleksei Buvailik, Prokhor Kotov, Roberto Calandra
Abstract
Humans can grasp everyday objects with almost perfect success rates using fingertip tactile feedback, yet much of the robotic grasping literature emphasizes vision-based grasp selection with parallel grippers. In this work, we systematically investigate how high-resolution, dynamic tactile sensing contributes to grasp stability prediction and model-guided grasping in dexterous robotic hands. To this end, we collected a dataset of 10,000 grasp trials across 200 objects using a multi-fingered robotic hand equipped with four Digit 360 tactile sensors, recording external vision, proprioception, and tactile streams throughout each grasp. With this dataset, we trained end-to-end temporal multimodal models to predict post-lift stability from pre-lift grasp observations and compared sensing modalities and encoding backbones. Experimental results and controlled input ablations show that incorporating touch, and particularly high-resolution, dynamic touch, improves grasp stability prediction. Finally, we deployed the learned predictor as an online stability gate on the real robot, where visuo-tactile model-guided regrasping improved the success rate among executed lifts by 10.5 percentage points over a non-tactile gate. These results show how rich fingertip sensing and expressive temporal models that capture the dynamics of touch can support learned grasping with multi-fingered hands without explicit contact or force modeling, providing a scalable data-driven path from tactile experience toward stable dexterous manipulation. The dataset is publicly available at https://lasr-lab.github.io/dexterous-grasp-stability/.