Prompt choice strongly affects voice cloning quality in zero-shot text to speech
What Makes a Good Audio Prompt for Zero-Shot TTS? Learning Prompt Selection from Synthesis Outcomes
Sound
Summary
Zero-shot text-to-speech models can copy a person’s voice using a short audio clip called a prompt. But picking which part of a longer recording to use as the prompt makes a big difference in how clear, natural, and similar the copied voice sounds. The authors studied many examples and found existing quality scores miss some important aspects like how easy it is to understand the speech. They created a small AI model that can pick better prompts by just listening to candidate audio segments, which improves clarity without needing to try speaking samples first.
What this means in practice
- •For voice assistant developers: Improve voice cloning clarity by automatically selecting the best audio segments without extra synthesis trials.
- •For audiobook producers: Choose optimal audio samples from long recordings to create more natural and intelligible synthetic voices for narrations.
Authors
Arnon Turetzky, Yossi Adi
Abstract
Zero-shot text-to-speech (TTS) models clone a target voice from a short reference clip, commonly referred to as the prompt. In practice, the prompt has to be cut from a longer recording of the speaker, so which part to use must be chosen. To understand what makes a good prompt, we analyze hundreds of thousands of outputs synthesized by four TTS models from candidate prompts of YouTube recordings. The choice of prompt substantially changes intelligibility, naturalness and speaker similarity. Standard predicted-quality scores capture mainly the naturalness part but miss most of the intelligibility gain. We find that the models largely agree on which prompt yields the highest measured speaker similarity, but much less on which prompt yields intelligible speech. We propose a small network that learns from the outputs a TTS model produced to rank a recording's candidate prompts from their audio alone. Without synthesizing anything at selection time, it picks prompts that lower the word error rate on held-out recordings by a third (Qwen3-TTS) to two fifths (F5-TTS) relative to a randomly chosen prompt, more than predicted quality achieves and nearly as much as synthesizing a dozen test sentences from every candidate first.