Robot uses visual cues to predict forces for better manipulation
Robot Learning with Visual Predicted Force
RoboticsArtificial Intelligence
Summary
Robots usually need special sensors to feel how hard they are pushing or pulling during tasks like picking up objects. The authors found a way for a robot to guess these forces just by looking at how its soft gripper bends. They teach the robot by showing it examples with known forces, then the robot learns to pick actions that create desired forces. This lets the robot handle delicate tasks like picking berries or inserting plugs without extra force sensors.
What this means in practice
- •For robotics engineers: Enable robots to perform careful contact tasks without installing force sensors by predicting forces visually from gripper shape changes.
- •For manufacturing automation teams: Improve robotic assembly processes by selecting actions based on visually predicted interaction forces, removing reliance on costly tactile hardware.
Authors
Haonan Chen, Feiyang Wu, Yuxiang Ma, Mustafa Mete, Pengfei Ye, Junxuan Shen, Cheng Zhu, Aurora Ruggeri, Kelvin Cheung, Jiayuan Mao, Edward Adelson, Jiajun Wu, Robert D. Howe, Yilun Du
Abstract
Force-aware manipulation typically relies on specialized force or tactile sensors. We show that force-aware manipulation can instead be achieved through visual force prediction from the deformation of a compliant Fin Ray gripper. Our approach trains two models. First, we train a visual force estimator on calibration data and use it to annotate task demonstrations with force estimates. Second, we train an action--force proposal policy on these force-augmented demonstrations to jointly generate candidate robot actions and their associated forces. At test time, we sample candidate actions and the forces they are expected to produce, then execute the action whose predicted force is closest to a target from the demonstrations. We evaluate our approach on berry picking, empty-can grasping, in-hand reorientation, and plug insertion. Our results show that visual force prediction can guide inference-time action selection for contact-rich manipulation without requiring force or tactile sensors at deployment.