Audio language model combines sound recognition with spatial awareness
MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness
Sound
Summary
Most audio-language models listen to sounds from just one microphone and miss important clues about where sounds come from. The authors created MiDashengLM-Spatial, a model that listens to sounds from multiple directions and understands both what the sound is and where it is coming from. They also built a system to create lots of training data with different spatial sound scenes to teach the model. Their model works well on tasks involving locating sounds as well as general sound understanding, without losing performance on usual sound recognition tasks.
What this means in practice
- •For audio engineers: Build tools that identify and locate multiple sounds in complex environments using one unified model.
- •For robotics developers: Enable robots with ears to understand what sounds occur nearby and where they come from for better interaction.
Authors
Jinbo Hu, Hang Su, Lichun Fan, Heinrich Dinkel, Gang Li, Zhanchen Dai, Yiru Zhang, Chang Liu, Peng Wang, Junnan Wu, Jian Luan, Cong Zou, Heng Qu
Abstract
Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.