Long speech prompts improve prosody reproduction in zero-shot text to speech
Long-Prompt Zero-Shot TTS: Benchmarking Prosody Reproduction Beyond Voice Cloning
Sound
Summary
Text-to-speech systems usually use short examples of a voice to copy the speaker's tone but struggle to capture how they naturally change pitch and emphasis. The authors test what happens when you give the system a much longer sample of someone's speech. They create a benchmark with many speakers and measures to see if the system can mimic timing, pitch, and emphasis better. Their results show that using longer speech prompts, up to about 10 minutes, helps the system make speech that sounds more like the original speaker’s natural rhythm and style.
What this means in practice
- •For voice assistant developers: Create more natural-sounding voice assistants by better capturing speaker-specific rhythm and emphasis from long speech samples without retraining.$Commercial implications: Enables voice assistant vendors to offer personalized voices with more accurate prosody reflecting real speakers, improving user experience and brand differentiation.
- •For audiobook producers: Improve audiobook narration quality by reproducing natural speaking styles and prosody from longer actor recordings without additional fine-tuning.
Authors
Dekun Chen, Renjie Xu, Yingda Shen, Qinke Ni, Zhizheng Wu
Abstract
Zero-shot text-to-speech (TTS) typically conditions on a short speech prompt, which conveys timbre but provides only limited evidence of a speaker's prosodic patterns. We formulate Long-Prompt Zero-Shot TTS, a task that uses long speech prompts at inference time to reproduce speaker-specific prosody without target-speaker fine-tuning. We introduce LP-TTS-Bench, a bilingual benchmark of 37 speakers with 63.95 prompt hours and 2,728 held-out test utterances. Beyond intelligibility and timbre similarity, LP-TTS-Bench evaluates timing, prominence, intonation, and model-based prosody metrics, enabling multidimensional evaluation of prosody reproduction. As a reference system, we develop LPTTS, whose hierarchical Long-Prompt Encoder compresses long speech prompts into a fixed number of continuous prosody vectors. Experiments show improvements in several prosodic dimensions over a short-prompt baseline, with overall gains as prompt duration increases up to approximately 10 minutes and saturation at longer durations.