Video helps computers learn touch sense from limited sensor data
Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision
Computer Vision and Pattern Recognition
Summary
Feeling touch and understanding physical contact is hard for computers and robots because actual touch sensors are expensive and limited. The authors created a method called Touch-JEPA that teaches computers to guess how hands touch objects by watching videos of hands and objects together. They start by letting the computer learn from videos alone without touch data, then improve it by showing a small amount of real touch sensor information. This approach helps computers better recognize when and how much contact happens, even for robotic hands without direct touch training.
What this means in practice
- •For robotic system designers: Equip robots with touch understanding by training on human hand videos without needing extensive tactile sensors on robots.
- •For augmented reality developers: Improve virtual object interaction realism by inferring touch contact and pressure from video of user hand movements.
Authors
Minseong Kweon, Qiao Feng, Yiming Huang, Lingjie Liu
Abstract
Understanding contact and tactile magnitude in hand-object interaction is a key step toward physically intelligent systems, yet learning such representations is limited by the cost and restricted coverage of tactile measurements. We present Touch-JEPA, a framework for learning tactile representations from hand-object videos with limited tactile supervision. We first introduce a self-supervised pretraining objective that learns from video alone without tactile supervision. Using video-derived hand geometry, Touch-JEPA predicts masked visual features at corresponding hand-surface locations from spatial and temporal context, encouraging the representation to capture latent contact dynamics. We then extend this formulation to joint representation learning, where masked prediction is applied to all videos while tactile supervision is available only for a labeled subset, grounding the shared representation in measured contact and magnitude. With self-supervised pretraining alone, simple supervised readouts on frozen Touch-JEPA features outperform supervised baselines in contact discrimination and tactile magnitude estimation across three human-hand benchmarks, and transfer to robotic hands without robot tactile training data. Joint learning with limited tactile supervision further improves contact and magnitude estimation on the human-hand benchmarks. Together, these results show that hand-object video can provide scalable self-supervision for tactile representation learning, while limited tactile measurements provide complementary physical grounding.