Audio visual ai models better identify speakers in conversations
Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs
Computer Vision and Pattern RecognitionSound
Summary
Understanding who is speaking in videos with multiple people talking is hard for current audio-visual AI models. The authors found that these models link sounds and visual parts using abstract symbols representing speech timing and speaker locations. They noticed that mistakes happen mostly when the audio and video don't line up well. To fix this, they used a simple method that marks active speakers with boxes using an existing detection tool, improving performance without extra training. A small amount of additional training further helps the models work better on other general audio-visual tasks.
What this means in practice
- •For video platform developers: Improve multi-speaker video understanding by accurately linking speakers to their speech using simple, training-free visual prompts.
- •For video conferencing software teams: Enhance transcription and speaker labeling accuracy in multi-person calls by applying lightweight fine-tuning after detecting active speakers visually.
Authors
Jihoo Jung, Youngjoon Jang, Joon Son Chung
Abstract
Current Audio-Visual LLMs (AVLLMs) struggle with reasoning over videos featuring multi-speaker dialogues. In such videos, resolving "who says what" is crucial, which necessitates trimodal (text-audio-visual) binding. Motivated by these challenges, we systematically investigate how this trimodal binding is achieved in AVLLMs. Specifically, we identify emergent symbolic trimodal binding mechanisms in AVLLMs that utilize modality-specific symbolic variables. By encoding auditory and visual components into symbolic variables-capturing temporal utterance sequences and spatial entity coordinates, respectively-the model establishes cross-modal linking within this abstract space. Crucially, we reveal that when trimodal binding fails, the breakdown predominantly stems from misaligned audio-visual connections. To overcome this bottleneck, we introduce an audio-visual prompting method utilizing an off-the-shelf Active Speaker Detection (ASD) model. By simply overlaying visual bounding boxes on active speakers, this training-free approach yields immediate performance gains across four conversation-centric benchmarks. Moreover, lightweight fine-tuning of fewer than 300 steps on these ASD-prompted-videos extends these gains to three general AV benchmarks, suggesting the generalizability of our method.