Multimodal learning improves soft robotic grasp recognition accuracy
SomaAlign: Reliability-Aware Multimodal Representation Learning for Soft Robotic Grasping
RoboticsMachine Learning
Summary
Soft robots must sense their environment using many types of sensors to grasp objects reliably, but each sensor has its own noise and errors. The authors created a large dataset called Soma with many objects and sensor readings to help machines learn better about grasping. They also developed SomaAlign, a method that combines these different sensor signals while being aware of their reliability, improving how robots understand grasping. This system can predict whether a robot’s grasp will succeed early on and recognize object features more accurately across different tasks.
What this means in practice
- •For soft robotics engineers: Improve robot grasp success prediction using combined vision, touch, and motion sensor data for more reliable manipulation.
- •For robotics system integrators: Use mixed sensor data representations learned from Soma to enable cross-modal retrieval and object attribute recognition in robotic systems.
Authors
Haoyuan Wang, Shuai Peng, Yongkai Liao, Yuxiao Sun, Wei Luo, Xinxing Chen, Jian Huang
Abstract
Robust multimodal perception for soft robotic grasping is challenged by modality-dependent noise, semantic ambiguity, and limited coordinated multisensory data. We introduce Soma, a dataset containing 100 objects, 1,000 continuous grasping trajectories, and 74,914 multimodal samples grouped by software acquisition cycle, spanning remote RGB, hand RGB, depth, tactile, IMU, and structured text observations. We also collect Soma-Success, an independent grasp-outcome dataset with 1,000 trials (998 retained after quality control), to evaluate frozen-encoder transfer for post-hoc grasp outcome classification and early grasp success prediction. Built on Soma, SomaAlign learns multimodal representations in three stages: masked reconstruction estimates relative modality reliability; reliability-aware contrastive alignment uses pair-specific adaptive temperatures to adjust alignment sharpness; and conflict-aware semantic constraints improve intra-sample cross-modal consistency while preserving global feature geometry. SomaAlign achieves an average Recall@10 of 90.66% across 30 directed cross-modal retrieval tasks. The learned representations also support physical attribute recognition, multimodal question answering, cross-modal generation, and cross-dataset transfer to grasp outcome prediction. On Soma-Success, frozen SomaAlign encoders with lightweight probes achieve 94.59% post-hoc classification accuracy and 84.07% early prediction accuracy. Together, Soma, Soma-Success, and SomaAlign provide a data and representation-learning foundation for robust and transferable multimodal perception in soft robotic manipulation.