Mask and replace training improves zero shot text to speech quality
Beyond Token Revision: Investigating Mask-and-Replace Diffusion for Zero-Shot Text-to-Speech
Sound
Summary
Generating speech from text without needing a lot of data or prior examples is hard. The authors studied a method where parts of a speech sequence are masked or replaced during training so the model learns to fix its own mistakes. They found that this training improves the accuracy of the speech output even if the model doesn't try to fix errors later. This suggests the benefit comes not just from error correction during generation, but from learning with noisy or changed inputs during training.
What this means in practice
- •For speech synthesis developers: Develop text-to-speech systems with better accuracy by using mask-and-replace training to improve token generation quality without relying on inference-time correction.
- •For voice assistant engineers: Implement more reliable zero-shot speech generation that generalizes better to new voices by leveraging training methods that tolerate noisy or changed inputs.
Tested on one dataset.
Authors
Hounsu Kim, Joonyong Park, Yuki Saito, Satoru Fukayama, Juhan Nam
Abstract
Unlike autoregressive models, discrete diffusion-based models for zero-shot text-to-speech generate speech tokens in parallel and can revisit earlier predictions. Mask-and-replace training extends mask-only training by randomly replacing some tokens, and its gains are commonly attributed to self-correction, the ability to revise previously generated tokens. However, exposure to randomly perturbed context during training may itself improve generation, raising the question of whether these gains require inference-time token revision. To investigate this question, we use DeMaR, which combines mask-and-replace training with confidence-ranked mask-only sampling while preserving the total training corruption probability. Trained from scratch on LibriTTS, DeMaR achieves lower word error rates (WER) than autoregressive and mask-only diffusion baselines using the same speech tokenizer. This advantage persists when each token remains unchanged after first being unmasked. Matched training conditions on two heterogeneous speech tokenizers show that both noisy-context augmentation and replacement supervision improve WER under this restriction. These findings demonstrate training-side benefits of replacement beyond enabling inference-time token revision.