Expert supervision affects text to speech quality in complex ways
Does Expert Supervision Improve Text-to-Speech? An Empirical Study of Preference Optimization
Sound
Summary
Improving text-to-speech (TTS) voices by learning from expert examples sounds promising, but it’s not clear if just copying better examples helps. The authors studied how using expert speech affects TTS models in different ways, revealing that better matching to expert voices doesn’t always mean better sounding or clearer speech. They found that supervision approaches can help avoid worse performance but don’t guarantee improvements beyond the original model. This shows that judging TTS improvements requires looking beyond just copying expert speech.
What this means in practice
- •For tts system developers: Optimize training strategies by choosing supervision methods that balance expert example quality and actual speech output improvements.
- •For voice assistant engineers: Understand limitations of expert-driven training to better manage quality and intelligibility in speech generated by assistants.
Authors
Minh Duc Vu, Junchuan Zhao, Ye Wang
Abstract
Preference optimization promises to improve text-to-speech (TTS) systems by learning which speech outputs are better. External expert speech seems a natural source of preferred responses, especially when student generations contain errors. Yet demonstration quality does not establish whether the student can benefit from that supervision, or whether better target fitting will produce better speech. We examine this gap across two TTS students using real and synthetic expert speech, comparing preference objectives and direct versus bridge supervision while tracking fixed-sequence likelihoods and generation metrics. With expert supervision held fixed, objectives differ markedly in their sensitivity to preference-score scaling and the resulting generation quality. Likelihood diagnostics characterize these responses, but cannot establish their benefit: expert likelihood can increase while intelligibility worsens, or decrease while it improves. Bridge supervision highlights a distinction between mitigating degradation and improving the student: it can recover performance lost under direct training without consistently surpassing the pretrained base. Route benefits vary across objectives, students, and expert sources; neither route consistently dominates. Together, these findings distinguish demonstration quality, target fitting, and generated-speech quality as complementary criteria for assessing expert supervision.