Post-training enables fine-grained natural language control of speech emotion and timing
Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language
Sound
Summary
Many text-to-speech systems can talk but have trouble making their voice change emotions or speaking speed in small parts of a sentence. This paper shows how to improve existing speech models by teaching them after they are built, so they can understand simple instructions about feeling and pace for different segments of speech. The researchers fine-tune these models using new techniques to make the changes accurate while keeping the person’s voice and words clear. This method does not need extra parts during speaking, making it easier to use.
What this means in practice
- •For audiobook producers: Create narrations that dynamically express emotions and pacing within sentences using simple text commands.$Commercial implications: Enables production companies to offer more expressive audiobook voices controlled by natural language prompts.
- •For virtual assistant developers: Add fine-tuned emotional and speed variations to virtual assistant responses for more natural and engaging conversations.
Authors
Lianru Gao, Yujie Guo, Yong Qin
Abstract
Audiobook narration, conversational agents, and audiovisual dubbing require speech that conveys changing emotions and adapts its pacing within a single utterance. But most existing TTS systems typically rely on utterance-level style conditioning, making such fine-grained control difficult to achieve. In light of this, and inspired by the success of post-training in large language models, we propose a unified post-training framework that equips pretrained text-to-speech models with natural-language control over segment-level emotion and duration. Supervised fine-tuning establishes instruction-conditioned speech generation, while reinforcement learning with group relative policy optimization refines control accuracy using emotion and duration rewards alongside content and speaker preservation objectives. By reusing the pretrained architecture, our approach avoids additional inference-time control modules. Experiments demonstrate significantly improved fine-grained controllability while maintaining speech intelligibility and speaker identity, highlighting post-training as a practical approach to extending existing speech synthesis models.