Touch sensing improves robot lifting success and force control
Compact Visuotactile World Models for Lifting: Prediction, Reward Alignment, and Force Constraints
RoboticsArtificial Intelligence
Summary
Robots need to predict when they will touch objects to grip and lift them effectively. The authors tested a model that uses both vision and touch to predict forces during lifting tasks in a simulator. They found that adding touch sensing greatly improves the accuracy of force predictions and helps the robot lift objects more successfully, especially when reward signals align with force constraints. However, controlling forces precisely while still completing the task remains challenging. Their tests were done in simulation and on recorded tactile data, without showing transfer to real robot systems.
What this means in practice
- •For robotics engineers: Design control systems for robotic hands that predict contact forces more accurately using combined vision and touch models.
- •For automation system integrators: Improve robotic lifting in manufacturing by aligning reward functions with physical force limits for safer object handling.
Tested on simulated data.
Authors
Qinzhen Ma, Sida Peng
Abstract
Accurate contact prediction is useful for robotic manipulation only if it supports effective decisions. We investigate this connection using a compact, randomly initialized visuotactile world model, trajectory-level uncertainty calibration, and behavior-initialized actor-critic learning in imagination. On 160 MuJoCo Lift episodes, adding touch reduces endpoint-force prediction error from 1.058 to 0.228 N and interval-peak error from 2.724 to 0.523 N across three training seeds. However, tactile persistence achieves lower errors of 0.095 and 0.498 N, respectively. Two exploratory control rounds comprise 680 executions on 40 independent test initial conditions. A matched reward revision on fresh test environments increases in-distribution 10 cm lifting success from 20.0% to 93.3%, while success within an 8 N per-finger budget reaches only 33.3%, compared with 70.0% for force feedback. Calibration margins reduce force violations at the cost of task completion. In a separate study of public GelSight recordings, a force regressor achieves 0.04234 N error, but frame-level calibration covers only 15.80% of complete trajectories; trajectory-level calibration raises this to 87.36% at nominal 90% coverage. Together, these findings distinguish improvements in sensing and task reward from improvements in force-constrained control. The evidence is limited to public sensing records and simulator execution, without a demonstrated transfer between them.