Audio-Visual Segmentation via Depth-Guided Collaborative Modeling
2026-08-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors study a task called Audio-Visual Segmentation (AVS), which means identifying and outlining objects in videos that make sounds by using both sight and sound information. They noticed that past methods often ignore the depth or distance of objects, which helps humans understand where sounds come from. To fix this, the authors added depth information as a third type of data and created a new system called DGCM-AVS that better separates objects that are close together. Their approach improved performance on a standard dataset, showing that depth is useful for this problem.
Audio-Visual Segmentationmultimodal perceptionpixel-level segmentationdepth estimationcross-modal alignmentdepth-aware dynamic modulatorprogressive fusionAVSS datasetsounding objects
Authors
Zhaojin Fu, Yuyang Hong, Qi Yang, Zili Wang, Kun Ding, Shiming Xiang, Bin Fan
Abstract
Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues. It has broad applications in video understanding, human-computer interaction, and autonomous driving. However, most existing AVS methods do not explicitly model geometric cues such as relative distance and occlusion, thereby limiting the robustness of cross-modal alignment. In human perception, spatial structure is naturally integrated with audio-visual evidence to accurately localize sounding objects. Motivated by this, we incorporate estimated depth as a spatial structural cue for AVS and propose DGCM-AVS, a tri-modal framework that jointly models audio, visual, and depth information. Specifically, we design a Depth-Aware Dynamic Modulator to improve the separation of adjacent objects while preserving intra-object feature consistency. Furthermore, we propose Depth-Guided Progressive Fusion, which uses depth as an intermediate bridge to progressively align audio cues with visual features. Compared to state-of-the-art methods, DGCM-AVS achieves relative improvements of 10.2 percent in M_J and 8.7 percent in M_F on the AVSS dataset. We believe our study highlights depth as a promising yet underexplored modality for AVS and may encourage further research in this direction.