Soundfield embeddings add spatial info to audio event detection
Bearings: Self-Supervised Soundfield Embeddings from First-Order Ambisonics
SoundArtificial Intelligence
Summary
Current systems that learn from audio scenes do not understand where sounds come from in space. The authors introduce Bearings, a way to teach computers to understand the spatial layout of sounds using special audio recordings called first-order Ambisonics. Bearings creates compact representations of sound fields that can combine with existing audio encoders without changing those encoders. This combination improves the ability to find and locate sounds in an environment, making detection much more accurate.
What this means in practice
- •For audio engineers: Enhance sound event detection systems with spatial awareness to improve locating sounds accurately in diverse environments.
- •For game developers: Integrate Bearings embeddings to add precise spatial sound detection, enabling more immersive and responsive game audio experiences.
Authors
Goksenin Yuksel, Marcel van Gerven, Kiki van der Heijden
Abstract
Recently proposed self-supervised audio encoders learn powerful general-purpose representations of sound scenes, yet they are spatially blind. To supply the missing spatial representation of sound scenes, we introduce Bearings. Bearings is a self-supervised framework that learns soundfield embeddings from unlabeled first-order Ambisonics. We pre-train a masked auto-encoder paired with a decoder conditioned on frozen acoustic embeddings from an off-the-shelf single-channel audio encoder. Our results show that the resulting soundfield embeddings form a reusable stream that can be attached to frozen acoustic encoders with a lightweight trainable fusion head. On sound event localization and detection, concatenating our soundfield embeddings with acoustic representations provides the missing spatial information and enables joint detection and localization, raising the location-dependent F-score from below 4 to 50 on TAU-NIGENS 2021 and 39 on STARSS23. To our knowledge, Bearings is the first self-supervised soundfield encoder whose embeddings plug into frozen acoustic encoders without retraining either model.