Distributed microphones improve acoustic scene understanding with geometry
Geometry-Informed Distributed Acoustic Scene Understanding
Sound
Summary
Understanding sounds across multiple rooms is hard because walls block sound signals. The authors propose a system that uses microphones placed in different rooms and combines their data using special neural networks and a language model aware of room layouts. This approach helps fill in missing information and creates a consistent story of what is happening in the space. Tests in simulated multi-room environments show this method works better than just using one microphone array.
What this means in practice
- •For smart home device makers: Create smart home systems that accurately interpret sounds from multiple rooms despite walls blocking signals.$Commercial implications: Enables multi-room audio understanding for smart assistants and security devices by combining distributed microphones and geometry-awareness.
- •For building automation teams: Improve indoor sound monitoring and event detection by fusing sound data across rooms to maintain spatial consistency.
Authors
Yiyuan Yang, Shitong Xu, Niki Trigoni, Andrew Markham
Abstract
Acoustic scene understanding in multi-room environments is a difficult task. Most existing systems use a single centralized microphone array, and they often fail because walls and doors block sound signals. To address this challenge, we propose a geometry-informed distributed acoustic scene understanding framework. Our system leverages distributed microphones and uses an audio spectrogram transformer and a topology-aware graph neural network to fuse spatio-temporal acoustic features. Then, these features are decoded into discrete semantic triplets. Finally, a frozen large language model combines these symbolic observations with the environmental geometry. This allows the system to perform spatial understanding, infer plausible missing transitions, and generate a physically consistent narrative of the scene. Experiments on a custom multi-room simulator demonstrate that our framework outperforms centralized baselines and improves spatial consistency under simulated occlusion.