ConformalMOS predicts speech quality with uncertainty intervals
ConformalMOS: Uncertainty-Aware MOS Prediction with Conformal Intervals and Ordinal Modeling
SoundComputation and Language
Summary
Evaluating how good synthetic speech sounds is important but hard because people’s judgments vary. The paper presents ConformalMOS, a method that not only predicts speech quality scores but also provides a range that likely contains the true score, showing the uncertainty. The authors improved the prediction by treating the scores as ordered categories rather than unrelated labels. This approach helps give more trustworthy estimates, especially when the speech is unusual or different from what the model has seen before.
What this means in practice
- •For speech synthesis developers: Provide reliable speech quality scores with confidence intervals to improve model selection and deployment decisions.
- •For voice assistant engineers: Use calibrated MOS estimates with uncertainty to monitor and improve synthetic voices in real-world applications.
Authors
Kehinde Elelu, Joshua E. Siegel, Mohammadali Saffary, Tashfain Ahmed, Simeon Babatunde, Ebuka Okpala
Abstract
Accurate prediction of human Mean Opinion Scores (MOS) is crucial for evaluating synthetic speech quality in text-to-speech (TTS) and voice conversion (VC). Most MOS predictors output only point estimates and ignore uncertainty, limiting reliability in model selection and deployment. We propose ConformalMOS, a framework that augments MOS prediction with conformal prediction to produce statistically valid prediction intervals with guaranteed coverage under exchangeability, alongside point estimates. During training, MOS labels are modeled in an ordinal-aware manner by converting one-hot targets into Gaussian-smoothed distributions, preserving score ordering. This improves uncertainty calibration, particularly when softmax confidence is overconfident on out-of-distribution speech. On BVCC datasets, ConformalMOS achieves competitive accuracy (MSE = 0.08) while delivering empirically validated coverage, enhancing reliability for real-world TTS and VC deployment.