Shift-aware method improves audio visual object tracking consistency
Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation
Computer Vision and Pattern Recognition
Summary
Identifying and precisely outlining objects in videos using sound, visuals, and language is tricky because the target can change position or become hidden temporarily. The authors introduce a new method that uses two types of internal markers to keep track of the object clearly, even when it’s partially invisible or the sound fades. Their method continuously updates memory to keep the object’s identity correct throughout the video. This makes identifying objects from combined audio, video, and language cues more accurate than previous methods.
What this means in practice
- •For video content analysis teams: Automatically segment and track specific objects referenced by spoken or textual descriptions more reliably in videos with complex audio-visual scenes.
- •For augmented reality developers: Improve persistent and accurate placement of virtual objects on real-world targets identified by audio-visual cues even when those targets move or are briefly obscured.
Authors
Kun Li, Sami Sebastian Brandt, Michael Ying Yang
Abstract
Referring Audio--Visual Segmentation (Ref-AVS) is an emergent multimodal task requiring the precise identification and segmentation of a specific object based on a natural language expression grounded in both visual and auditory cues. Unlike traditional AVS, which segments generic sound sources, or referring video object segmentation (Ref-VOS), which relies solely on visual--textual grounding, Ref-AVS necessitates fine-grained tri-modal interaction across vision, audio, and language. However, existing methods often fail to maintain consistency when subjected to target shift, a phenomenon where the model erroneously transfers its focus between objects due to transient occlusions, acoustic fading, or identity ambiguities. To address this, we propose a novel Shift-Aware Identity-guided Latent (SAIL) refinement framework. It introduces a dual-token strategy that decouples frame-level dynamics from a global semantic anchor to explicitly model target-shift cues. We design a shift-aware refinement module that rectifies drift by aligning tokens with domain-specific evidence. Finally, we leverage a decoder that integrates the refined embeddings with a continuous memory update to maintain spatio-temporal coherence and identity consistency throughout the video sequence. Extensive experiments conducted on two Ref-AVS benchmarks demonstrate the effectiveness of our proposed method, significantly outperforming existing AVS, Ref-VOS, and Ref-AVS approaches. The code will be released after publication.