Semantic acoustic imaging maps sound sources in 3D space
SAID: Semantic Acoustic Imaging Detector for Sound Event Localization and Detection
Sound
Summary
People can hear sounds around them and tell what they are and where they come from. The authors created a way for computers to do the same by turning sounds into detailed images that show exactly where each sound is in every direction around you. Their system can separate overlapping sounds and label them, like footsteps or music, on these sound maps. They trained their method using simulated and real recordings, and it performed best on a recent sound localization challenge.
What this means in practice
- •For robotics developers: Equip robots to identify and locate multiple sound sources as spatial regions with classes, improving environmental awareness.
- •For augmented reality app developers: Create AR applications that visually display the locations and types of sounds around users in real time as semantic acoustic maps.
Authors
Runbang Wang, Zining Liang, Yin Cao, Qiuqiang Kong
Abstract
In daily life, people hear speech, footsteps, and music around them. We can often recognize these sounds and judge where they come from. Each sound source can be shown on a separate acoustic map, a rectangular image covering $360^{\circ}$ horizontally and $180^{\circ}$ vertically. The map shows the directions occupied by the source as a region and the sound energy within that region. A class label identifies the sound. Predicting these labeled acoustic maps from audio is called semantic acoustic imaging. Such maps could help robots perceive their surroundings and allow augmented reality displays to show sound regions and classes over the real world. Existing models can recognize sound classes and estimate a direction for each source. However, a direction alone does not describe the source region or its energy. Acoustic imaging must also distinguish sound sources in nearby directions, while the number of active sources and the regions they occupy can change over time. We therefore propose the Semantic Acoustic Imaging Detector (SAID), which predicts a separate labeled acoustic map for each active source from audio. First, we pretrain Audio2Sph, SAID's audio encoder, through sound energy estimation across directions without class labels. Then, we train the complete SAID model to predict source regions, energy, and classes together. We also develop a pipeline that generates simulated recordings for pretraining and supports fine-tuning on real recordings. On the official DCASE2026 Task 3 Track A evaluation set, our submitted system ranks first with 0.1080 macro-averaged mean average precision (Macro mAP) and 0.3962 Macro Pearson $r$. Demos and code are provided at https://github.com/IN03X/SAID.