Unified alignment improves robot perception across multiple sensor types

UniOMA: Unified Optimal-Transport Multi-Modal Structural Alignment for Robot Perception

RoboticsArtificial Intelligence

Summary

Robots often need to understand information that comes from very different sensors, like touch or force sensors, but making sense of all these signals together is tricky. The authors designed a new way to better align these different types of sensor data by looking not just at individual matching parts, but at the overall structure and relationships in the data. Their method helps robots make sense of complex environments more reliably by combining data in a more organized way. They tested their approach on several robot tasks and found it works better than previous methods.

What this means in practice

  • For robotic perception engineers: Improve multisensor data integration by aligning structural relationships among diverse sensor signals including tactile and force data.
  • For robot control developers: Enhance prediction of robot and environment states by using unified geometric alignment of multimodal sensory inputs.

Authors

Xinrui Zu, Mark Hoogendoorn, Kevin Sebastian Luck, Shujian Yu

Abstract

Achieving generalizable and well-aligned multimodal representations remains a core challenge in robot learning. While recent approaches have attempted to align modalities by modeling conditional or higher-order statistical dependencies, they do not explicitly consider the structural coherence across modalities. In this work, we propose a novel multimodal alignment method that augments existing contrastive losses with a geometry-aware Gromov--Wasserstein (GW) distance-based regularization. To this end, we encode intra-modal geometry for each modality and reduce cross-modal structural discrepancies through a dynamically learned GW barycenter, thereby encouraging structural consistency in addition to instance-level contrastive alignment. We apply this optimal-transport-based alignment strategy to robot perception tasks involving underexplored modalities such as force and tactile signals, where modality data often exhibit varying sample densities. Extensive experiments on five robotic benchmarks show that our method improves structural alignment and downstream performance over state-of-the-art multimodal contrastive baselines across a wide range of downstream tasks, including robot and environment state prediction. Moreover, our GW-based regularization can be seamlessly integrated into most InfoNCE-like objectives.