Spatial audio improves tracking of sound-producing people in videos
AVPS-360: A Spatial Audio-Visual Benchmark for Instance Segmentation and Tracking of Sound-Producing Persons
Computer Vision and Pattern Recognition
Summary
Tracking who is making sounds in videos is hard when only regular audio from one microphone is used. The authors created a large new collection of videos that use special 360-degree sound recording to better locate where sounds come from. They also made a new method that uses this spatial sound together with video images to better identify and follow people making sounds like talking or clapping. Their tests show that older methods don’t work well on these long videos with breaks in sound, but the new method does better at both finding and linking the correct people to their sounds.
What this means in practice
- •For video analytics teams: Segment and track individual people making sounds in 360-degree video footage for detailed behavior analysis.
- •For augmented reality developers: Integrate spatial audio-visual cues to improve real-time tracking and identification of people producing sounds in immersive environments.$Commercial implications: Enables AR products to accurately identify and track sound sources in 3D space, enhancing interactive experiences.
Authors
Yifu Sun, Parthasaarathy Sudarsanam, Tuomas Virtanen, Archontis Politis
Abstract
Audio-visual instance segmentation (AVIS) segments and tracks individual sound-producing objects in videos. Existing benchmarks predominantly use monaural audio, which lacks inter-channel cues for spatially distinguishing concurrent sound sources. We introduce AVPS-360 (Audio-Visual Person Segmentation 360), a spatial audio-visual benchmark bridging AVIS and sound event localisation and detection (SELD). Built upon STARSS23, it pairs first-order Ambisonics audio with synchronised 360° video across 230 recordings totalling 10.5 hours. Recordings average 164.0 seconds. The dataset captures intermittent sound production in real-world interactions dominated by speech, alongside laughter, clapping, and footsteps. It provides 31,678 manually reviewed instance masks linked to source identities and sound-event labels. We also present a baseline that integrates spatial audio with visual features for frame-level segmentation and identity association across frames. Experiments on AVPS-360 show that existing AVIS methods struggle with detection and identity association in long recordings with intermittent sound production, while our baseline substantially improves performance in both aspects.