Perceptually Better, Semantically Worse: Measuring Speech Enhancement Impact on LLM-Based Voice Systems
2026-08-31 • Sound
Sound
AI summaryⓘ
The authors studied if improving speech quality with enhancement tools always helps large language models (LLMs) understand spoken commands better. They created a measure called Output Divergence Rate (ODR) to see how often these enhancements change the LLM’s intended meaning compared to clean speech. Their tests showed that all speech enhancement methods caused some changes in LLM understanding, sometimes even making it worse despite better audio quality scores. This indicates that common audio quality measures do not reliably predict how speech enhancement affects LLM tasks. The authors conclude that better evaluation methods are needed for speech preprocessing in AI pipelines.
Speech EnhancementLarge Language Models (LLMs)Output Divergence Rate (ODR)Intent ClassificationAudio Quality MetricsMetricGAN+Whisper modelwav2vec2-largeAutomatic Speech Recognition (ASR)SLURP dataset
Authors
Randy Frans Fela, Pejman Mowlaee
Abstract
Speech enhancement (SE) is commonly applied as a preprocessing step in spoken AI pipelines under the assumption that better audio quality improves downstream task performance. Whether SE-induced distortions propagate to downstream LLM task performance remains an open question. We introduce Output Divergence Rate (ODR), which measures how often SE changes an LLM's intent classification relative to clean speech, and benchmark five conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades. Every condition produces ODR significantly above zero ($p < 0.001$, binomial test). MetricGAN{+} more than doubles ODR versus unenhanced noisy speech (0.318 vs. 0.135) despite improving PESQ, and unmitigated echo reaches an ODR of 0.836 through speaker substitution, a failure WER cannot capture. Audio quality metrics range from near-zero to moderate correlation with ODR (SQUIM-MOS $ρ=-0.068$, PESQ $ρ=-0.467$). The MetricGAN{+} and echo results replicate across ASR architectures, indicating that standard audio quality metrics are insufficient for LLM pipeline quality.