Distributed microphones improve sound scene understanding across rooms
Geometry-Informed Distributed Acoustic Scene Understanding
Sound
Summary
Understanding sounds in multiple rooms is tricky because walls block noises from traveling clearly. To tackle this, the authors created a system that uses microphones placed in different rooms instead of just one spot. Their system combines sound patterns collected over time and space with information about the layout of the building to better understand what is happening. It can guess sounds that might be hidden and tells a consistent story about the whole environment. Tests show that this approach works better than traditional single-microphone setups, especially when some sounds are blocked.
Acoustic scene understandingDistributed microphonesAudio spectrogram transformerGraph neural networkSpatio-temporal featuresSemantic tripletsLarge language modelEnvironmental geometrySpatial consistencyOcclusion
Authors
Yiyuan Yang, Shitong Xu, Niki Trigoni, Andrew Markham
Abstract
Acoustic scene understanding in multi-room environments is a difficult task. Most existing systems use a single centralized microphone array, and they often fail because walls and doors block sound signals. To address this challenge, we propose a geometry-informed distributed acoustic scene understanding framework. Our system leverages distributed microphones and uses an audio spectrogram transformer and a topology-aware graph neural network to fuse spatio-temporal acoustic features. Then, these features are decoded into discrete semantic triplets. Finally, a frozen large language model combines these symbolic observations with the environmental geometry. This allows the system to perform spatial understanding, infer plausible missing transitions, and generate a physically consistent narrative of the scene. Experiments on a custom multi-room simulator demonstrate that our framework outperforms centralized baselines and improves spatial consistency under simulated occlusion.