Audio visual segmentation improves object separation with competitive slots

SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention

Computer Vision and Pattern Recognition

Summary

Audio-Visual Segmentation means figuring out which parts of a video show the objects that are making sounds. The authors found that when many sound sources are present, usual methods mix the objects together or get confused. They made a new way called SlotAVS that separates the sounds and images by letting different parts of their model compete to claim objects, which keeps things clearer. Their approach helps computers better recognize and outline multiple sounding objects in videos.

What this means in practice

  • For video processing engineers: Build video editing tools that automatically isolate and mask multiple sound-producing objects in complex scenes for targeted effects or analysis.
  • For robotics developers: Enable robots to better identify and localize individual sound sources visually in cluttered environments for improved interaction and navigation.

Authors

Arnav Sharma, Anand Mishra, Renu Rameshan

Abstract

Audio-Visual Segmentation (AVS) requires predicting pixel-wise masks of sounding objects by grounding acoustic signals in visual space. In multi-source scenes, concurrent sound sources often remain entangled within shared visual representations, leading to merged masks or unstable specialization when using standard alignment-focused transformer architectures. To address this, we present SlotAVS, an object-centric framework driven by competitive bi-modal slot attention. While robust foundation models provide rich semantic priors, SlotAVS shifts the paradigm from global cross-modal fusion to explicit competitive binding. By initializing discrete object slots from temporal audio sequences and enforcing strict slot-wise competition over spatial visual tokens, our method structurally mitigates the blending of distinct acoustic entities. Further, to stabilize instance-level learning, we introduce a Hungarian matching strategy that optimally aligns slots with ground-truth objects using spatial and boundary overlap. Coupled with a numerically stable, confidence-aware mask fusion mechanism, SlotAVS robustly aggregates multi-source predictions. Extensive experiments on the AVSBench dataset demonstrate the efficacy of our approach, achieving 88.6% mIoU on the Single-Source (S4) split and 71.7% mIoU on the Multi-Source (MS3) split. These results show that explicit competitive binding is critical for complex audio-visual scene understanding and effectively leveraging foundation-model priors without relying solely on their capacity.