Audio visual system isolates sounds by type in 360 degree videos
AV-QTSS: Class-Query Target Sound Spatialization in Multi-Source 360$^\circ$ Videos
Sound
Summary
When watching 360-degree videos, it is important for sounds to come from the right directions to feel real. The authors created a method that, given a mixed sound recording, a 360-degree video, and the type of sound you want, can isolate and place that sound in the correct direction around you. They use a special map to guess where each sound source is based on the video and improve how it sounds in space. Their system works better than previous methods especially when many sounds happen at once.
What this means in practice
- •For virtual reality developers: Create immersive VR scenes that isolate and spatialize specific sound types from 360-degree videos to enhance user experience.
- •For audio post-production teams: Improve mixing by extracting and positioning individual sound sources from multi-source 360-degree recordings for clearer audio edits.
Authors
Qingyu Luo, Peng Zhang, Wenwu Wang, Philip J. B. Jackson
Abstract
Spatial audio is central to immersive 360$^\circ$ media, yet existing audio-visual spatialization renders the scene as a whole, leaving no control over an individual source. We introduce Audio-Visual Query-conditioned Target Sound Spatialization (AV-QTSS): given a monaural multi-source mixture, a 360$^\circ$ video and a class query from the 13-class DCASE sound-event ontology, it generates the first-order Ambisonics (FOA) signal of the queried source alone. The omnidirectional mixture contains no explicit spatial cues, so direction has to be inferred from the video, where several objects may sound at once. We propose a query-conditioned dense binding localizer that predicts a spherical probability map over the whole scene, coupled to the renderer so both are optimized jointly. Experiments with one to four concurrent sources show improved localization and spatial accuracy relative to established methods and ablated variants, and the localizer transfers to real 360$^\circ$ recordings after adaptation.