Haikus spoken with better rhythm and emotion in speech synthesis
HaikuS2S: A Cascaded System For Responding In Verse
Computation and LanguageArtificial IntelligenceSound
Summary
Making computers speak poetry like haikus is hard because haikus have a special 5-7-5 syllable rhythm and pauses at the end of lines. This paper presents a system called HaikuS2S that first listens to speech, then uses a smart language model to create haikus, and finally adjusts the way the computer speaks them. The system was trained on normal poetry and also special haiku examples to improve how natural and expressive the speech sounds. The authors found that their method helps the computer match the rhythm and tones of haikus better while keeping the emotional feel consistent.
What this means in practice
- •For voice assistant developers: Integrate haiku-style responses with correct rhythm and pauses to enhance poetic speech in conversational AI.
- •For interactive entertainment designers: Create characters or virtual narrators that recite haikus naturally with improved emotional and rhythmic alignment.
Authors
Devangi Sharma, Sophia Judicke, Glenda Tan, Conrad Schaumburg, Shinji Watanabe
Abstract
Expressive speech synthesis has advanced through prosody modeling, yet generating structured poetic speech, such as haiku, remains challenging. Prior work on prosody transfer improves expressiveness, and fine-tuned poetry TTS (text-to-speech) systems capture verse intonation. However, these models do not model haiku's 5-7-5 syllable structure or line-ending pauses. We present a cascaded system, HaikuS2S, combining ASR (automatic speech recognition), LLM (large language model)-generated haiku, and TTS fine-tuning on both prose and custom haiku datasets. Our evaluation focuses on emotion similarity, speech quality, and prosody alignment. In our experiments, we see that our prosody and tonal alignment improve significantly with our fine-tuned systems, particularly the one trained on both general poetry and haiku. We also see that we maintain similar emotion similarity scores across all systems.