Speech judges overweight loudness and emotion without clear reasoning
Louder, Longer, Livelier: Acoustic Shortcuts and Underspecified Rationales in Speech LLM Judges
Sound
Summary
Evaluating spoken language with large language models (LLMs) adds challenges because these models must consider sounds as well as words. The authors found that these speech-model judges often give higher scores simply based on louder volume or emotional tone, even when these cues don’t actually indicate better quality. They also found the models’ explanations rarely mention the actual sound clues influencing their decisions, making their reasons unclear. This means speech LLM judges need to better focus on relevant audio features and explain their choices based on those features.
What this means in practice
- •For speech technology developers: Identify and reduce misleading acoustic biases in speech quality evaluation models to better align automated scoring with human perception.
- •For customer service analytics teams: Use improved speech quality assessments that avoid overvaluing loudness or emotion, enhancing evaluation of call center interactions.
Authors
Mingyue Huo, Shivam Mehta, Bhavin Jawade, Yinghong Lan, Haoqi Li
Abstract
LLM-as-a-judge is widely used for evaluating text, but extending this paradigm to speech requires models to interpret acoustic as well as linguistic evidence. This introduces a modality-specific risk: a speech judge may treat a perceptually salient cue as evidence of quality even when that cue is irrelevant to the target criterion or receives more weight than human listeners give it. We call this behavior an acoustic shortcut. To study it, we audit six speech LLM judges using controlled manipulations of intensity, content richness, and emotional delivery. We evaluate both pointwise scoring and pairwise comparison, using human preference calibration to interpret the results. The judges consistently reward louder audio, prefer content-rich speech more strongly than human listeners do, and map emotional delivery into quality preferences. These effects are most visible in pairwise comparison, while pointwise scores often obscure them. More concerningly, the accompanying rationales rarely identify the acoustic cue that changes a judgment and instead repeatedly rely on a limited vocabulary, leaving them acoustically underspecified. Together, these findings show that reliable speech judges must both resist acoustic shortcuts and ground their rationales in the acoustic evidence behind their decisions. To support reproducibility and future audits, we also release SpeechJudgeAudit, the controlled stimuli and evaluation tools used in this study.