Prompt-consistency inference improves zero-shot speech synthesis quality
Prompt-Consistency Inference for Zero-Shot Flow-Matching Text-to-Speech Models
Sound
Summary
Generating speech that sounds like a particular speaker without training on their voice is hard. The paper shows that by keeping the initial example speech consistent during the generation steps, the system produces speech that sounds more like the original speaker and is easier to understand. This method doesn’t need extra training or slower processing, just a smarter step during generation. The authors tested this on flow-matching models and found it improves quality reliably.
What this means in practice
- •For voice assistant developers: Improve voice assistants’ ability to mimic new speakers without additional training data by applying prompt-consistency during speech generation.
- •For call center software teams: Enhance synthesized speech quality for customer service agents by reducing speaker mismatch in zero-shot TTS systems.
Authors
Vasily Zadorozhnyy, Can Goksen, Kazuhito Koishida, Dung Tran
Abstract
In recent years, flow-matching models have produced significant improvements in zero-shot text-to-speech synthesis. Conditioned on an audio prompt and text, these models learn a velocity field and generate speech by iteratively solving an ODE. During inference, the solver evolves a single state spanning both the prompt and the region to be generated, although only the generated region is ultimately retained. The discarded prompt state, however, still matters; its intermediate values influence generation through the velocity field that couples the two regions. As sampling continues, this state can drift away from the prescribed conditional path, introducing a discrepancy into subsequent generation updates. Unlike the unknown generated trajectory, the prompt path is available in closed form from the reference audio and the initial noise. We exploit this observation with Prompt-Consistency Inference (PCI), a training-free rule that restores the prompt block to its analytic value before each velocity evaluation, while leaving the generated block unchanged. PCI improves speaker similarity and intelligibility across the evaluated flow-matching TTS backbones without additional network evaluations. Our ablation studies further show that PCI keeps post-step prompt discrepancies smaller and that corrections covering the later sampling stages recover much of the observed similarity gain.