HEAR Who Said What: Unlocking Speaker-Attributed Reasoning via Counterfactual Voice Grounding
Computation and LanguageArtificial IntelligenceSound
Summary
The authors created a test called HEAR to check how well speech models can tell who is speaking in recordings with multiple people. They found that many top speech models have trouble doing this and often guess based on the words instead of the voices. To improve this, they made a new model called A2R, trained with special audio data that teaches it to focus on voice sounds rather than just language. This new model did better on HEAR and could handle new tasks without extra training. They shared all their data and models for others to use.
Speech Language ModelsSpeaker AttributionMulti-Speaker AudioHierarchical BenchmarkAcoustic Vocal CuesSemantic PriorsCounterfactual AudioZero-Shot GeneralizationAudio DatasetsSpeaker-Level Hard Negatives
Authors
Dongwook Lee, Sangkwon Park, Eunwoo Song, Che Hyun Lee, Youngho Cho, Junho Kim, June Young Yi, Heeseung Kim, Sungroh Yoon
Abstract
Speech Language Models (SLMs) are increasingly deployed in multi-speaker environments, yet their ability to attribute speech to the correct speaker and reason over speaker identities remains unclear. Hence, we introduce HEAR, a conceptually hierarchical benchmark diagnosing the foundational capabilities of speaker-attributed reasoning, comprising 2.4K human-verified samples from 887 diverse multi-party audio clips. Evaluating 20 leading SLMs on HEAR reveals they struggle with these foundational tasks, often relying on semantic priors rather than actual vocal cues. To address this, we present A2R, a 30B model optimized on Counterfactual Audio with Speaker-level Hard negatives (CASH), a dataset designed to guide the model to prioritize acoustic vocal cues over linguistic signals. A2R achieves strong performance on HEAR and exhibits zero-shot generalization to diverse multi-speaker downstream tasks, demonstrating that learned speaker attribution unlocks the model's latent capacity for speaker-aware reasoning. All resources are available at https://attributetoreason.github.io/AttributeToReason/