Lightweight models cut size with minor performance loss in sound scene analysis
Lightweight Systems for Spatial Semantic Segmentation of Sound Scenes
Sound
Summary
Separating and labeling sounds coming from different directions usually requires large computer models. The authors studied if smaller, simpler models could do the same job while fitting on small devices. They tested different tagging and sound separation models and measured how well they performed compared to bigger versions. The results showed that lightweight models can still do an acceptable job with less computing power.
What this means in practice
- •For embedded system developers: Implement spatial audio scene analysis on devices with limited memory and processing power using lightweight tagging and separation models.
- •For smart home device engineers: Enable multichannel sound source detection and classification with smaller models that fit in home assistants and IoT audio devices.
Authors
Mayank Mishra, Paul Magron, Romain Serizel
Abstract
Spatial semantic segmentation of sound scenes (S5) jointly performs audio source separation and tagging on a multichannel audio mixture. Therefore, an S5 system can be decomposed into an audio tagging model and a separation model. The baseline S5 system in task~4 of the DCASE 2025 challenge uses M2D and ResUNet models for tagging and separation, respectively. These are large in overall size and therefore not appropriate for use in embedded devices. The main aim of this paper is to assess the performance loss in an S5 system when using lightweight tagging and separation models instead. To this end, we first compare the tagging performance of lightweight audio tagging models, namely PANNs and DyMNs, in terms of label prediction accuracy against the baseline M2D model. Secondly, we assess separation performance in terms of the scale-invariant signal-to-distortion ratio of the baseline ResUNet separator relative to that of a convolutional-based separator using SuDoRMRF. Finally, we combine the tagging and separation models across various configurations and assess the overall performance of the S5 system using the class-aware signal-to-distortion ratio.