Audio visual segmentation improves by aligning sounds with image regions

Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation

Computer Vision and Pattern Recognition

Summary

Audio-Visual Segmentation (AVS) is about finding and outlining objects in videos that are making sounds by looking at both the pictures and the sounds together. The authors found that previous methods had trouble because the audio and visual information often don’t match well or get mixed up. They created a new technique that better aligns the sounds with the right parts of the image using contrastive learning and prototypes that represent categories. Their method performs better than others on common tests and works well even in different situations.

What this means in practice

Authors

Junbo Zhang, Hang Su, Zhaofan Li, Hang Dong, Chao Sun

Abstract

Audio-Visual Segmentation (AVS) aims to accurately segment sounding objects in video frames by leveraging audio-visual correspondence cues. However, it remains challenging due to the intrinsic semantic incompleteness within a single modality and the semantic gap between audio and visual representations. Traditional feature-fusion decoding approaches struggle to suppress fusion noise effectively, while recent methods that incorporate data-dependent priors often increase the complexity of modeling audio-visual correlations, leading to poor cross-domain generalization. To address these issues, we propose a novel adaptive contrastive and prototype learning framework, BYOAVP, for AVS. Specifically, we design a Self-Supervised Audio Enhancement (SSAE) module that introduces contrastive learning to adaptively align audio representations with gradient-blocked visual embeddings, thus narrowing the semantic gap between modalities. Furthermore, a Dynamic Prototype Constraint (DPC) module is developed to refine pixel-wise category perception via momentum-based prototype updating, while enhancing localization of sounding regions through cross-modal interaction. Extensive experiments demonstrate that our method achieves SOTA performance across two AVS benchmarks and six sub-tasks, exhibiting strong robustness and generalization ability.