Tactile sensing improves robot touch tasks across varied conditions
TacMind: Benchmarking and Modeling Tactile Interaction States for Robotic Manipulation
Robotics
Summary
Robots often need to touch and handle things carefully, but it's hard to compare how well their sense of touch works in different tasks. The authors created a large set of robot touch tasks in simulation to test this. They also developed a method that helps robots remember and predict what their sense of touch will feel like in the future, which improves how they decide to act. Their tests showed that using touch information consistently helped robots perform better on these tasks.
What this means in practice
- •For robotics engineers: Build robots that handle objects precisely under varied touch and movement conditions using the new benchmark and tactile prediction models.
- •For robot software developers: Improve robot manipulation software to better predict and adapt to future tactile sensations, enhancing touch-based control.
Authors
Jiaxin Shi, Zibin Wu, Yichi Zhang, Jianghua Duan, Siyu Zhu, Weihao Yuan
Abstract
Tactile sensing provides robots with direct observations of contact physics, local geometry, and interaction dynamics, making it a vital source of feedback for contact-rich manipulation. However, existing tactile manipulation studies are often developed and evaluated within specific task settings, making it difficult to systematically compare how tactile information contributes under different interaction conditions. In this work, we develop a benchmark in simulation comprising 128 tasks organized around three complementary evaluation regimes: Precise, Recognition, and Dynamic. The benchmark further includes controlled static and dynamic task variants and 12 perturbation settings for evaluating robustness under changes in visual and physical conditions. These settings motivate studying tactile information as a temporally evolving interaction representation rather than only as an instantaneous observation. We therefore propose an action-conditioned latent predictive world model that integrates tactile history and learns a tactile state through prediction of future tactile latents. Building on this representation, we design a model that conditions action generation on the learned tactile state and uses predicted future tactile information to refine candidate actions through predictive tactile reflection. Experiments on the proposed benchmark show consistent gains from incorporating tactile information across the evaluated regimes. Project Page: https://tacmind-bench.github.io/