Speech rewarded style planning improves text to speech emotion and style control
Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech
SoundComputation and Language
Summary
Making computer speech sound more natural and expressive is hard because turning style instructions into actual speech sounds doesn't always work as expected. The researchers found that simply matching text descriptions to speech doesn't ensure the speech will have the right style. They created a new way called Speech-Rewarded Style Planning that learns to pick better style instructions by rewarding those that make the speech sound closer to a target. This method improved how well a speech system matched the desired emotion and style when tested on a special conversation dataset.
What this means in practice
- •For speech synthesis engineers: Improve expressive style control in conversational TTS systems to generate speech that better matches target emotions and styles.
- •For virtual assistant developers: Generate more naturally styled and emotionally appropriate spoken responses in dialogue systems by optimizing style instructions with speech-based rewards.
Tested on one dataset.
Authors
Shiao Zhu, Lianbo Liu, Sizhen Lyu, Yuzhe Wang, Sheng Li, Takahiro Shinozaki
Abstract
Natural-language style descriptions provide an interpretable interface between large language models (LLMs) and controllable text-to-speech (TTS). However, using descriptions as pseudo-labels compresses target acoustics into text, and descriptive fidelity need not imply effective control of a particular synthesizer. We empirically show that speech-text alignment only weakly predicts downstream acoustic similarity among candidate instructions for the same utterance. We therefore propose Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner through a frozen downstream TTS model. Given dialogue history and response text, the planner generates candidate instructions and is optimized with group-relative policy optimization (GRPO), using the teacher-forced likelihood of target speech tokens as the reward. On an English subset of the ISCSLP 2026 CoT-TTS corpus, SRSP achieves higher speech-style and emotion similarity to target speech and lower mel-cepstral distortion than the Base LLM and target-audio-informed captioning baselines. LLM-based expressive speech evaluation further shows gains over all baselines in contextual appropriateness and reference consistency.