Audio visual segmentation improves by aligning sounds with image regions
Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation
Computer Vision and Pattern Recognition
Summary
Audio-Visual Segmentation (AVS) is about finding and outlining objects in videos that are making sounds by looking at both the pictures and the sounds together. The authors found that previous methods had trouble because the audio and visual information often don’t match well or get mixed up. They created a new technique that better aligns the sounds with the right parts of the image using contrastive learning and prototypes that represent categories. Their method performs better than others on common tests and works well even in different situations.
What this means in practice
- •For video analysis developers: Improve tools that automatically detect and segment sound-making objects in videos by better matching audio and visual data.
- •For assistive technology engineers: Enhance devices that rely on identifying sound sources visually, such as aid for hearing-impaired users or smart surveillance.
Authors
Junbo Zhang, Hang Su, Zhaofan Li, Hang Dong, Chao Sun
Abstract
Audio-Visual Segmentation (AVS) aims to accurately segment sounding objects in video frames by leveraging audio-visual correspondence cues. However, it remains challenging due to the intrinsic semantic incompleteness within a single modality and the semantic gap between audio and visual representations. Traditional feature-fusion decoding approaches struggle to suppress fusion noise effectively, while recent methods that incorporate data-dependent priors often increase the complexity of modeling audio-visual correlations, leading to poor cross-domain generalization. To address these issues, we propose a novel adaptive contrastive and prototype learning framework, BYOAVP, for AVS. Specifically, we design a Self-Supervised Audio Enhancement (SSAE) module that introduces contrastive learning to adaptively align audio representations with gradient-blocked visual embeddings, thus narrowing the semantic gap between modalities. Furthermore, a Dynamic Prototype Constraint (DPC) module is developed to refine pixel-wise category perception via momentum-based prototype updating, while enhancing localization of sounding regions through cross-modal interaction. Extensive experiments demonstrate that our method achieves SOTA performance across two AVS benchmarks and six sub-tasks, exhibiting strong robustness and generalization ability.