Summary
Figuring out who is speaking about what in audio recordings is usually difficult for audio language models, which often perform worse than chance when asked to match speakers to topics. The authors found that by adjusting internal attention settings in the model—without extra training—they could reliably steer the model to describe any chosen speaker with high accuracy. Surprisingly, the attention patterns that do this well mostly come from text language models, not audio-specific ones, and transferring these patterns works surprisingly well. The team also studied different ways to find these crucial attention patterns and found that some methods are better than others at ensuring helpful outputs.
What this means in practice
- •For voice assistant developers: Enhance audio assistants to focus on specific speakers by reusing attention patterns from text models without extra audio training.
- •For audio analytics teams: Improve speaker-related query handling in multi-speaker recordings by adjusting attention weights derived from text-based models.
Abstract
Asked to describe what one of six speakers in a recording talks about, audio language models describe the right one on 6 to 16% of trials, below the 16.7% a guess would give. Adding a fixed bias to the attention logits of a hundred heads, under a tenth of the model's and with no training, redirects the description to whichever speaker we choose, on 90.7% to 99.0% of trials. Those heads are largely not specific to audio. Rank the text-only language model an audio model was built from, or a released model of the same family, on a written version of the task, take its top hundred heads, and carry them over unchanged: they redirect the audio model on 80.8% to 95.0% of trials, with nothing about audio entering the selection. The audio and text head sets share 66 to 74 of 100 where chance would give about 20, and the shared part alone reproduces almost all of the steering. What that does not show is that sharing is what makes the heads work: an equal-sized draw from the same discovered hundred does nearly as well, and none of our three models separates the two explanations. A second finding concerns how such heads are found. Ranking heads by how much attention they place on the segment asked about, as an established score does, or by how much of their attention moves with the question, as a per-head normalised variant does, gives top hundreds that share 69, 37 and 4 heads across our three models. In Ultravox, where they share 4, the established score's heads leave output the judge cannot place on any segment on 69.7% of trials, against 40.0% with no intervention and 1.0% for the normalised variant. That is one arm of six; on the other five the established score steers above a random draw.