Tactohand predicts detailed hand touch and forces from video
TactoHand: Learning Geometry-Based Dense Hand-Surface Interaction Fields from Video
Computer Vision and Pattern Recognition
Summary
Understanding how hands touch and interact with objects is important for many technologies, but capturing this in detail usually needs special sensors. The authors created TactoHand, a computer program that looks at regular video and guesses where the hand touches or nears objects, and how much force it applies. They trained it on a huge video collection using smart methods to label when and how hands contact objects. This helps computers better understand hand movements without needing complex hardware.
What this means in practice
- •For augmented reality developers: Use video input to estimate detailed hand contacts and forces for realistic hand interaction in AR experiences.
- •For robotic grasping engineers: Improve hand-object interaction models by predicting contact and force fields without needing tactile sensors.
Authors
Ziheng Xi, Weibo Li, Jiayi Yu, Sipeng Zheng, Wanpeng Zhang, Haiweng Xu, Hao Luo, Chuan Mao, Zongqing Lu
Abstract
Understanding how humans interact with the world requires a representation of physical interactions at the hand surface. Tactile sensing captures these interactions, but collecting tactile data at scale requires specialized hardware. We introduce TactoHand, a model that predicts three dense hand-surface interaction fields---contact, proximity, and 3D force---from monocular video. We derive geometry-based contact and proximity labels and motion-derived force labels from eligible training data within a curated collection spanning approximately 50 million source frames. TactoHand uses a temporal transformer to aggregate visual evidence across frames, while three learned queries decode the corresponding hand-surface fields. A differentiable physics layer further constrains force predictions using observed object motion. Across in-domain and out-of-domain benchmarks, TactoHand improves contact and proximity estimation over prior methods. In addition, physics supervision reduces position and rotation errors in motion rollouts, and the predicted contact and proximity fields improve downstream hand-object reconstruction. Together, these results show a scalable, low-cost path to learning hand-surface interaction fields without collecting tactile measurements. Code will be released at https://github.com/zhenqis123/Tactohand.