Human data helps robots learn touch and motion in physical tasks
ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
Robotics
Summary
Robots often struggle to learn how to handle objects when touching and moving them because collecting this data via robot control is slow and limited. The authors show it is possible to train a system to predict how touch and motion behave using only data from humans manipulating sensors by hand. This human data helps the system learn contact dynamics that carry over to robots better than the visual look of scenes. So, robots might better understand how objects feel and move by learning from people instead of just their own limited experience.
What this means in practice
- •For robotic manipulation engineers: Build robot systems that predict future tactile and motion states by training on human physical interaction data, improving understanding of contact during manipulation.
- •For industrial automation teams: Enhance robotic assembly lines by transferring human touch dynamics learning to robots for better handling of objects across varied contact tasks.
Authors
Yuxiang Ma, Weihan Xu, Pengfei Ye, Anna Mei, Yuxin Ray Song, Tingle Li, Haonan Chen, Edward H. Adelson
Abstract
Abstract Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.