Instruction supervision improves text to speech quality with less error
Stabilizing Instruction Supervision for Instruct-TTS via Controllable Diversification and Drift Filtering
Sound
Summary
Text-to-speech (TTS) systems use instructions written by large language models to guide how speech sounds, but many of these instructions can change the meaning and make the speech less accurate. The authors found that over 40% of these rewritten instructions had problems, which weakened the speech’s quality. They developed a new method that carefully expands instructions, filters out bad rewrites, and aligns speech features with the instructions, making the instructions more reliable. Their approach improved how well the system follows instructions and reduced errors in rewritten instructions significantly.
text-to-speechinstruction supervisionsemantic driftlarge language modelprosody controldata-centric approachfine-tuninginstruction diversificationquality control
Authors
Yizhong Geng, Kecan Mao, Qifei Li, Cong Wang, Yingming Gao, Ruimin Wang, Chunfeng Wang, Hao Li, Ya Li
Abstract
Instruct-TTS systems expand structured style labels into natural-language training instructions through LLM rewriting, yet we find that over 40% of unconstrained rewrites contain semantic drift that corrupts supervision and weakens generalization. We formalize this problem as instruction supervision instability and propose a data-centric stabilization recipe that jointly improves coverage and fidelity through three mechanisms: controllable instruction diversification for systematic expansion, LLM-based drift filtering for quality control, and attribute-aligned supervision that grounds prosody control in acoustic perturbations. On the Chinese split of InstructTTSEval, our recipe raises instruction-following from 34.5% without fine-tuning and 51.0% with naive fine-tuning to 56.4%, while constrained rewriting reduces drift from 40.4% to 15.4%. Ablations confirm the three mechanisms are complementary, and the drift taxonomy may generalize to instruction-driven generation beyond TTS.