EditVoice enables flexible zero-shot text to speech and speech editing
EditVoice: Variable-Length Non-Autoregressive Zero-Shot TTS and Speech Editing with Edit Flows
Sound
Summary
Making computer-generated speech usually requires knowing how long the speech will be before creating it. The authors introduce EditVoice, a new system that can change both the words and the length of speech while it's being made. It can add, remove, or change parts of the speech based on text input and can work with partial speech examples given before or after the new text. EditVoice can also improve or edit recorded speech, showing it works well even beyond its original training examples.
What this means in practice
- •For speech technology developers: Generate or edit speech dynamically without preset length constraints in zero-shot text-to-speech applications.
- •For audio post-production teams: Perform seamless text-based edits on recorded or generated speech for refining audio content without re-recording.
Authors
Hongyao Deng, Wenhao Guan, Xuetao Lin, Peijie Chen, Weijie Wu, Lin Li, Qingyang Hong
Abstract
Recent non-autoregressive (NAR) zero-shot text-to-speech (TTS) models generate in parallel but typically require the target sequence length to be specified before generation. We introduce EditVoice, to our knowledge the first variable-length NAR zero-shot TTS model, which uses Edit Flows to jointly update speech content and sequence length through insertions, deletions, and substitutions. EditVoice adopts speech-infilling training, which unifies zero-shot TTS and text-based speech editing and allows both prefix and suffix speech prompt placements at inference. We introduce Complementary Prompt Sampling (CPS) to leverage the complementary Edit Flow predictions induced by the two prompt placements. We further find that EditVoice can edit source and model-generated speech beyond its training sources. We use this generalization for end-to-end editing and training-free post-generation refinement. With the Edit Flow model trained on 10K h of GigaSpeech, EditVoice demonstrates competitive zero-shot TTS performance on Seed-TTS Eval EN and LibriSpeech-PC and speech editing performance on RealEdit.