Cross-lingual voice cloning works without transcripts or forced alignment
Cross-Lingual F5-TTS 2: A Simplified Framework for Language-Agnostic Voice Cloning
Sound
Summary
Making a computer talk like someone else by hearing just a short clip is tricky, especially when the computer doesn’t have a written transcript of that clip. The researchers found a simpler way to train these systems without needing accurate timing information between speech and text. They use an existing model to create training pairs automatically, making it easier and more reliable to clone voices in many languages. Their new method copies speakers’ voices more closely and still sounds clear, even without knowing what was originally said.
What this means in practice
- •For voice assistant developers: Produce cross-lingual voice clones without needing text transcripts or forced alignment, simplifying multilingual voice personalization.
- •For audiobook producers: Create consistent voice clones from brief audio samples even for languages lacking text transcripts, aiding rapid multilingual audiobook generation.
Authors
Qingyu Liu, Rixi Xu, Yushen Chen, Zhikang Niu, Haitao Li, Pengcheng Zhu, Bowen Zhang, Jian Zhao, Yunting Yang, Qinyuan Cheng, Xipeng Qiu, Berrak Sisman, Kai Yu, Xie Chen
Abstract
Zero-shot text-to-speech (TTS) can clone a speaker's voice from a short audio prompt, yet most TTS systems still require the audio prompt transcript during inference. This dependency prevents cross-lingual voice cloning when the audio prompt transcript is unavailable, particularly for unseen languages. Cross-Lingual F5-TTS removes this dependency and enables transcript-free cross-lingual voice cloning, but it prepares its training data with forced alignment. Forced alignment is sensitive to boundary errors, and its cost grows as more languages are covered. Its speaking rate predictor is also unreliable at estimating duration when the audio prompt begins or ends with silence. In this paper, we present Cross-Lingual F5-TTS 2, a simplified framework for transcript-free cross-lingual voice cloning without forced alignment. Instead of using forced alignment to segment real utterances, we build same-speaker prompt and target pairs using a pretrained F5-TTS model and fine-tune the same model on these constructed pairs. This simplifies data preparation and preserves the acoustic modeling capability of the pretrained model, enabling adaptation with only a short fine-tuning stage. We further make the syllable-level speaking rate predictor robust to leading and trailing silence through silence-aware augmentation. Experiments show that Cross-Lingual F5-TTS 2 reaches higher speaker similarity than F5-TTS and Cross-Lingual F5-TTS while maintaining intelligibility. All related resources are publicly available.