AudioSphere creates self-supervised 3D audio models for events and localization
AudioSphere: Towards Self-Supervised Spatial Audio Representation Models
SoundArtificial Intelligence
Summary
Most audio AI models learn sounds from one microphone but miss how sounds come from different directions. The authors show it’s possible to teach a computer to understand both what and where sounds are happening using recordings that capture how sound moves in space. They built AudioSphere, a system trained to fill in missing pieces of 3D sound recordings so it learns detailed sound and direction info. This system works well for recognizing specific sound events and figuring out their locations, even without extra training.
What this means in practice
- •For audio engineers: Use frozen AudioSphere models to detect and locate sounds accurately in multi-microphone recordings without needing new labeled data.
- •For sound system designers: Integrate AudioSphere features to improve real-time spatial audio recognition systems for immersive environments or smart devices.
Authors
Goksenin Yuksel, Marcel van Gerven, Kiki van der Heijden
Abstract
Recent audio embedding models learn representations that transfer across clip- and frame-level tasks, yet they operate almost exclusively on single-channel signals and are blind to spatial structure. Spatial audio models capture that structure but rely on clip-level supervision or contrastive objectives, yielding embeddings unsuitable for frame-level tasks like sound event detection and localization (SELD). We argue this dichotomy is not fundamental: patch-level spatial self-supervision can jointly learn acoustic and spatial features at both temporal granularities. We introduce AudioSphere, a multi-channel masked autoencoder trained on Ambisonics recordings that reconstructs both spectral content and directional cues via a joint spatiotemporal masking strategy that masks both time-frequency patches and intensity vectors. AudioSphere supports frame-level predictions while producing strong clip-level representations via aggregation. Alongside the model, we release RealSELD, a standardized benchmark for evaluating frozen backbones on real-world multi-channel SELD. AudioSphere enables, for the first time, SELD with a frozen self-supervised backbone, while matching or exceeding prior self-supervised audio embedding models on clip-level classification tasks spanning environmental sound, speech, and music, and on frame-level sound event detection. All code and data are available at https://github.com/labhamlet/AudioSphere and https://huggingface.co/labhamlet.