Object centric audio representation enhances multiple sound tasks

Self-Supervised Object-Centric Representation for Multi-Task Audio Analysis

Sound

Summary

Understanding sounds in a complex audio mix can be tricky. The authors show that representing sounds as separate objects—like pieces in a puzzle—helps computers recognize, detect, and separate different sounds better. They trained their system on fake audio mixes so it could learn how to break down sounds into parts without needing labeled examples. This approach works well across several tasks, suggesting it could generalize to many real-world audio problems.

What this means in practice

  • For audio software developers: Develop audio tools that identify, detect, and separate overlapping sounds using a unified object-based representation for improved accuracy.
  • For hearing aid designers: Incorporate object-centric audio processing to improve hearing aids’ ability to separate and highlight sounds from noisy environments.

Authors

Parthasaarathy Sudarsanam, Tuomas Virtanen

Abstract

This paper studies the capability of an object-centric audio representation to support multiple downstream tasks within a unified framework. The object-centric model is trained in a self-supervised setup using slot attention by reconstructing Audio Spectrogram Transformer (AST) patch embeddings from a set of latent slots. It is trained on synthetic audio mixtures, which provides controlled multi-source audio scenes. These slot representations are evaluated for the classification, detection, and separation of sound events by training task specific output layers. On the classification task, the model achieves an F1-score of 0.8, comparable to clip-level AST embeddings which achieve 0.7. For sound event detection, it reaches a macro F1-score of 0.7, outperforming clip-level AST embeddings at 0.4 and matching AST patch embeddings at 0.7, which encode finer time-frequency information. In source separation, it achieves an SI-SDR improvement (SI-SDRi) of 18.4 dB. Furthermore, in a zero-shot source separation setting, the model achieves an SI-SDRi of 14.5 dB. These results demonstrate that slot-based representations capture transferable and compositional structure in audio scenes, enabling generalization across tasks and datasets.