Unified model improves robot understanding from multiple senses
Beyond Omni: A Unified MLLM for Embodied Sensory Understanding
Computer Vision and Pattern Recognition
Summary
It is hard for machines to understand the world using different types of senses beyond just sight and sound. The authors created a system called Unisense that connects many kinds of sensory information, like touch, smell, and motion, to language so a machine can better interpret them. They trained Unisense on a large collection of examples with eight different sense types. Tests show it performs better than other models on many tasks involving these senses, helping machines understand their environment in richer ways.
What this means in practice
- •For robotics engineers: Enable robots to interpret diverse sensory inputs like touch, smell, and motion through a single unified model to improve environment understanding and interaction.
- •For autonomous vehicle developers: Enhance vehicle perception systems by integrating multiple sensor modalities such as radar, thermal, and point clouds within a shared language-based framework for better decision-making.
Authors
Shiyun Xiong, Dongming Wu, Peiwen Sun, Bokang Yang, Jiaxu Wang, Xiaolou Sun, Mingxing Wen, Xiangyu Yue
Abstract
Recent embodied systems increasingly build on multimodal large language models (MLLMs) for their pretrained understanding and reasoning capabilities, but their perception of the physical world remains largely limited to basic modalities such as vision and audio. However, broadening MLLMs to more diverse sensory modalities from the physical environment is challenging: their representations are often weakly connected to the MLLM and vary substantially in structure and statistics, while their interpretation still requires shared high-level understanding and reasoning. We therefore propose Unisense, a unified embodied MLLM framework that supports diverse physical sensory modalities within a shared multimodal architecture. Unisense follows a two-stage training process: (i) aligning sensory representations with language to establish a language-accessible interface; and (ii) tuning modality-specific experts within the shared LLM to provide specialized processing for different sensory modalities. To support broad sensory learning, we curate Unisense-1M, a large-scale embodied sensory training corpus comprising 1M samples across 8 sensory modalities. We further evaluate Unisense on 12 benchmarks spanning depth, point cloud, IMU, tactile, thermal, radar, motion, and smell understanding. Across these heterogeneous sensing tasks, Unisense achieves strong performance and outperforms both generalist and specialized models on most benchmarks, demonstrating the potential of pretrained MLLMs as unified models for embodied sensory understanding.