Spatial audio benchmark measures assistant’s real home sound understanding
RMS-AQA: A Two-Stage Spatial Audio Question Answering Benchmark for Real-World Domestic Environments
Sound
Summary
Understanding sounds in a home is hard because noises come from different places and times. The authors created a test called RMS-AQA that checks if computer systems can listen to sounds in a house, figure out where and when they happened, and answer questions about them. They use both real sound recordings and realistic computer-made sounds to make the test more like everyday life. Their experiments show that the biggest challenges are sounds coming from many sources at once, from far away, and differences between real and synthetic sounds.
What this means in practice
- •For smart home device developers: Improve voice assistants’ ability to understand where and when sounds occur in home environments using spatial audio data.$Commercial implications: Enables more context-aware and accurate smart home assistants that respond to sound events location-specifically, improving user experience.
- •For robotic navigation engineers: Use the dataset and benchmark to train robots to locate and interpret sounds in real indoor spaces for better interaction and safety.
Authors
Peihao Chen, Qing Wang, Lichun Fan, Yufeng Hao, Zhifeng Kong, Mengyao Zhu, Hengyi Hong, Hang Chen, Hang Su, Yujie Jian, Chao-Han Huck Yang, Shichao Hu, Jun Du, Jian Luan, Ke Li
Abstract
Embodied assistants in domestic environments must infer what happened, where and when it occurred, and how to respond. To address this, we introduce RMS-AQA, a spatial audio question answering (SAQA) benchmark for real-world domestic environments. The benchmark features a two-stage question-answering (QA) format to comprehensively assess the ability of audio-language models (ALMs) to first ground audible sound events and subsequently perform complex spatio-temporal reasoning based on that grounding. To maximize acoustic realism, our dataset combines authentic real-world first-order Ambisonics (FOA) recordings with high-fidelity synthetic data generated using measured room impulse responses (RIRs). Furthermore, we provide a lightweight spatial plug-in that injects FOA-format data into frozen audio-language backbones. Experimental results reveal that the primary challenges stem from concurrent sources, far distance, and sim-to-real domain gap between RIR-synthesized and authentic recordings.