Hierarchical audio models improve multitask sound and scene recognition
Hierarchical Interpretable Multitask Audio Representation Learning: An Analysis of Event-Scene Relationships through the Information Bottleneck
Artificial Intelligence
Summary
Sound recognition models usually try to identify events and scenes separately, which misses how sounds naturally fit into environments. The authors tested a new approach where the model first guesses individual sounds and then uses that information to predict the overall scene. They found this method helps the model learn both tasks together better and creates clearer, more meaningful internal representations. Their approach also helps explain how different sounds relate to different environments.
What this means in practice
- •For audio engineers: Build audio recognition systems that better link individual sounds to their environments for improved multitask performance.
- •For smart home device designers: Integrate hierarchical audio models to improve scene understanding and event detection in smart assistants and IoT devices.
Authors
Claudia Montero-Ramírez, Alba Martínez-Serrano, Carmen Peláez-Moreno
Abstract
Joint Acoustic Scene Classification (ASC) and Audio Tagging (AT) models traditionally rely on unconstrained multihead architectures. However, this design ignores the natural hierarchical dependency between sounds and acoustic scenes. In this work, we investigate a hierarchical (sequential) conditioning paradigm that predicts scene categories directly from intermediate event probabilities. Through a large-scale empirical benchmark with 66 full training runs across three state-of-the-art backbones (PaSST, PANNs, and BEATs) and different loss-weighting settings (beta in [0.0, 1.0]), we show that hierarchical conditioning achieves competitive classification performance while altering multitask optimization dynamics. Unlike multihead baselines, which sequentially solve the easier ASC task before optimizing AT, the hierarchical architecture enables synchronized dual-task learning for specific loss-weighting settings (beta in [0.6, 0.9]). Furthermore, we propose a multi-perspective interpretability framework that combines an analysis of the information bottleneck, manifold geometry, and Formal Concept Analysis (FCA) on a Term-Frequency Inverse-Document-Frequency (TF-IDF) event representation, providing complementary insights into event-scene relationships. Our findings reveal that hierarchical conditioning systematically yields lower conditional entropy, higher mutual information, and superior latent cluster separability (higher Silhouette scores), establishing a more structured and semantically faithful intermediate feature space.