Refinement and depth improve different parts of text to speech quality
Refinement Buys Intelligibility, Search Buys Identity: What Test-Time Compute Buys in Masked-Diffusion TTS
Artificial Intelligence
Summary
Making computer voices sound natural involves two main tricks: making the model bigger and letting it refine its guess more times when speaking. This paper shows these two tricks do different jobs: refining improves how clear the speech is, while bigger models help keep the speaker's unique voice. The researchers found that trying harder with one method can’t fully replace the other and that most speaker identity issues come from a part called the codec. So, to make voices sound clear and true to a person, both model size and refinement steps need to be considered separately.
What this means in practice
- •For voice assistant developers: Tune text-to-speech systems by balancing model size and refinement steps to separately improve clarity and speaker recognition in synthesized speech.
- •For audiobook production teams: Improve the quality of synthetic narrations by optimizing refinement and model depth for better speech intelligibility and speaker identity preservation.
Authors
Nityanand Mathur, Hamees Sayed, Ayush Pratap Singh
Abstract
Diffusion language models for text-to-speech combine two forms of computation: model depth (parameters) and refinement steps (inference budget). We ask whether they scale equally across capabilities. We train 15 masked-diffusion codec TTS models varying depth (19-133M parameters, 3 seeds) on 2,000 hours of speech and sweep refinement steps T in [1,16] at inference, measuring zero-shot synthesis via ASR word error rate (intelligibility) and speaker verification (identity) on 174 held-out speakers. Against measured floors, refinement closes 86.2% of the intelligibility range but only 46.4% of the identity range - a 1.86x asymmetry robust across multiple error metrics. Retraining at 3x and 6x schedule attenuates but does not reverse this gap (1.84 to 1.36 to 1.23x), because intelligibility saturates with steps while identity continues improving. Best-of-K search recovers speaker identity where refinement fails, with 64.6-79.0% win rates across four independent encoders. Depth and steps are not interchangeable: separable B(d)B(T) fits significantly better (Delta AICc=+69.3) than substitution models. Analysis shows 62% of remaining identity deficit lies in the codec, not the generator. We conclude that refinement and depth target different bottlenecks and should be optimized separately.