Closed-loop guidance reduces accents in cross-language voice cloning
Closed-Loop Dual-Reference Guidance for Accent Control in Cross-Lingual Voice Cloning
SoundComputation and LanguageMachine Learning
Summary
Voice cloning technology can copy someone's voice but often keeps their original accent even when speaking another language. The authors propose a method using two voice references: the original and a specially converted version that sounds native in the target language. This helps generate speech with less of the original accent while keeping the voice similar. Tests show their approach improves pronunciation and speaker identity while requiring less computing power.
What this means in practice
- •For voice assistant developers: Generate voices for assistants that maintain user voice identity but reduce accent in different languages, improving naturalness and clarity.$Commercial implications: Enables commercial multilingual voice assistants with user voice personalization and clearer pronunciation for diverse users.
- •For call center technology teams: Improve text-to-speech synthesis for multilingual customer support voices that sound more native and retain brand identity across languages.
Authors
Yuanzhong Zheng, Wei Huang, Ming Ma, Kailun Jin, Youhai Jiang
Abstract
Zero-shot cross-lingual voice cloning often carries the reference speaker's accent into the target language. We propose closed-loop dual-reference guidance, which pairs the original reference with a reusable native anchor constructed by voice conversion (VC). The references initialize separate language-model (LM) states that guide a shared acoustic trajectory; each completed patch then updates both states, keeping their generated histories aligned with the emitted speech. Anchor construction runs once per speaker, and online synthesis uses a frozen TTS model without per-utterance VC. On 20 held-out French speakers, the fixed setting $λ^{\!*}=0.33$ reduces residual French LID mass by 73\% versus direct cloning, at an ECAPA similarity cost of only $0.042$; relative to anchor-only cloning, it recovers $0.155$. The method achieves the lowest mean WER ($0.009$) and highest mean GOP ($4.750$) among the held-out test arms. In blinded preference tests, native-English listeners favor it over direct cloning on pronunciation naturalness, and over both anchor-only cloning and a TTS$\to$VC cascade on speaker identity. Online RTF is approximately 26\% lower than the cascade's ($0.59$ vs.\ $0.80$), and the same setting reduces Spanish LID mass by 56\% without re-tuning.