Plan-to-speech failures linked to persona sentiment conflicts studied
When the Persona Contradicts the Line: Diagnosing Plan-to-Synthesis Failures in Chain-of-Thought TTS
Computation and LanguageSound
Summary
Sometimes, text-to-speech systems struggle to express the right emotions when a character's personality conflicts with the sentiment of the words they say. The authors created a test in both Japanese and English to study these conflicts and where the system fails. They found that planners usually set the right emotional direction, but the final speech often doesn't match perfectly. By improving the way the system connects planning to speech generation, they increased the accuracy of emotional delivery. Their results vary based on language and different speech features.
What this means in practice
- •For voice assistant developers: Improve emotional expression in assistants when users specify conflicting personality and sentiment to avoid unnatural speech output.
- •For multilingual tts system builders: Use the bilingual benchmark and improved planning-to-synthesis methods to better control emotional speech in Japanese and English voice synthesis.
Authors
Joonyong Park, Jerry Li
Abstract
Chain-of-thought TTS systems rely on language-model planners to translate persona and context into vocal delivery, but it is unclear where this intent is lost when the requested affect conflicts with lexical sentiment. We introduce a bilingual Japanese--English persona--line conflict benchmark in which each item pairs one line with congruent and conflicting personas and pre-declares the expected acoustic direction. We probe this direction in the planner-generated delivery phrase, the planner-written emotion vector, the QwenEmotion-generated vector, and the rendered waveform. Planner outputs are usually direction-correct, while downstream agreement can be substantially lower. Bypassing QwenEmotion provides only modest gains over the full planner-controlled Base. Supervising QwenEmotion raises the direction rate to 75% in Japanese and 84% in English. A learned text-to-latent interface reaches the highest English rate of 89% and the highest Japanese arousal rate of 88%. These results show that learned repair of the plan-to-synthesis interface can recover counter-textual delivery, although its effect differs across languages and acoustic axes.