Predicted speech quality scores often disagree with human preferences
How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?
Sound
Summary
People often use predicted scores called MOS to judge how good speech enhancement methods sound, but these scores don’t always match what humans prefer. The authors introduce a new way to measure how well these predicted scores agree with human choices when picking better systems. They found that even the best prediction models still disagree with humans about a quarter of the time. Improving predictions by combining models or adapting them to new conditions helps only a little, especially for unknown speakers or systems.
What this means in practice
- •For speech enhancement developers: Evaluate and select speech enhancement methods more reliably by considering system-level preference accuracy instead of relying solely on predicted MOS correlations.
- •For voice communication engineers: Improve quality assessment practices by recognizing the limitations of automated speech quality predictors for unseen speakers or systems in real deployments.
Authors
Nahomi Kusunoki, Tsubasa Ochiai, Naohiro Tawara, Marc Delcroix, Naoyuki Kamo, Tetsuji Ogawa, Shoko Araki
Abstract
We investigate whether predicted Mean Opinion Scores (MOS) can reliably support system-level comparisons of speech enhancement (SE) methods by introducing system-level preference accuracy (SPA). Although MOS prediction models are widely used to evaluate SE systems, their performance is typically assessed by correlation with human-rated MOS, which does not guarantee agreement on which system is better. SPA addresses this gap by directly evaluating whether predicted and human-rated MOS yield the same system preferences. Using SPA, we systematically evaluate three settings: single prediction models, ensembling, and domain adaptation. Through experiments, SPA varies substantially across single prediction models, from 9.4% to 76.8%. Even the best model disagrees with human judgments in approximately 23% of system comparisons. Ensembling yields only limited improvement, while domain adaptation tends to substantially improve SPA in the closed condition but brings only modest gains in the more practical open condition, where neither the target systems nor the speakers are known. These results suggest that SPA can reveal errors correlation-based evaluation alone does not expose, and that predicted MOS alone can lead to unreliable conclusions in practical SE system comparison.