Instruction supervision stability improves talking AI voice style control
Stabilizing Instruction Supervision for Instruct-TTS via Controllable Diversification and Drift Filtering
Sound
Summary
Spoken text-to-speech systems that follow style instructions can have trouble understanding those instructions correctly. The authors found that when instructions are rewritten by large language models, more than 40% changed the meaning too much. They fixed this by diversifying instructions in a controlled way, filtering out bad rewrites using AI, and aligning voice features with these instructions. This approach helped the system follow instructions better and made the voice sound more natural.
What this means in practice
- •For speech synthesis developers: Improve voice assistants and TTS systems by stabilizing style instructions for better natural-sounding speech control.
- •For voice user interface designers: Create more reliable prosody and style control in conversational agents using filtered and diversified instruction data.
Authors
Yizhong Geng, Kecan Mao, Qifei Li, Cong Wang, Yingming Gao, Ruimin Wang, Chunfeng Wang, Hao Li, Ya Li
Abstract
Instruct-TTS systems expand structured style labels into natural-language training instructions through LLM rewriting, yet we find that over 40% of unconstrained rewrites contain semantic drift that corrupts supervision and weakens generalization. We formalize this problem as instruction supervision instability and propose a data-centric stabilization recipe that jointly improves coverage and fidelity through three mechanisms: controllable instruction diversification for systematic expansion, LLM-based drift filtering for quality control, and attribute-aligned supervision that grounds prosody control in acoustic perturbations. On the Chinese split of InstructTTSEval, our recipe raises instruction-following from 34.5% without fine-tuning and 51.0% with naive fine-tuning to 56.4%, while constrained rewriting reduces drift from 40.4% to 15.4%. Ablations confirm the three mechanisms are complementary, and the drift taxonomy may generalize to instruction-driven generation beyond TTS.