Force torque signals enable robots to detect object contact changes
Language-Grounded Force/Torque Pretraining for Extrinsic Contact Perception
Robotics
Summary
Robots need to know when the things they hold touch other objects, but cameras can't always see this well. The authors show that using force and torque sensors can help robots understand these contacts better. They trained a system by linking sensor data to language descriptions using simulated actions. This approach lets robots identify when objects touch or support each other and quickly learn new tasks like plugging in connectors.
What this means in practice
- •For robotics engineers: Enable robots to detect and interpret object contact events using pretrained force/torque sensor data linked to language representations.
- •For automation system designers: Improve adaptation to novel manipulation tasks like connector insertion by efficiently fine-tuning force/torque based contact perception models.
Authors
Kazuki Takahashi, Hiroaki Murakami, Ryota Sakuma, Mitsuhiro Kamezaki, Matthew Ishige, Yoshihiro Kawahara
Abstract
Fine manipulation requires robots to recognize when a grasped object makes or changes contact with its environment, yet such extrinsic contact is not always directly observable from vision or local contact geometry. Force and torque signals carry this information, but also vary with object weight, grasp configuration, and robot motion, leaving it unclear whether they can provide a reusable representation of physical state. Here we show that bilateral six-axis force/torque histories can be pretrained as a semantically structured physical representation by aligning them with a frozen language space using supervision generated from simulated pick-and-place interactions. On unseen objects in simulation, the learned representation supports text-prototype readout of robot action, object weight, and grasp interaction. In real-world placement, unlabeled calibration enables contact and support-state recognition without updating the encoder, outperforming a vertical-force sign-reversal rule baseline. For connector insertion, an event absent from pretraining, low-rank adaptation improves data-efficient learning over training from scratch. These results show that force/torque histories can support both direct readout of known physical states and efficient adaptation to new extrinsic-contact events.