Voice cloning improves accuracy and speaker identity without ratio tuning
RAWD-TTS: Ratio-Free Reward Alignment for Discrete-Diffusion Voice Cloning
Sound
Summary
Making computers talk exactly like someone from just a short voice clip usually has trouble getting the words right and keeping the speaker's unique voice. The authors propose a new method called RAWD-TTS that fine-tunes the speech output by giving rewards based on how well the words match and how much the voice sounds like the original speaker. This method helps the computer produce clearer, more accurate speech that still sounds like the person, all without needing complex calculations about how the speech is generated. They tested it on Russian speech samples and found it reduced errors and better matched the speaker's voice.
What this means in practice
- •For speech technology teams: Improve voice cloning systems to produce clearer speech that better preserves speaker identity without complex tuning.
- •For call center software developers: Build personalized synthetic voices that more accurately match individual speakers from short voice recordings.$Commercial implications: Enables commercial voice cloning products offering high-quality, personalized voice synthesis for customer service applications.
Authors
Maxim Maslov, Kirill Borodin, Vasilii Kudryavtsev, Nikita Vasiliev, Grach Mkrtchian
Abstract
Zero-shot text-to-speech synthesizes new utterances in a speaker's voice from a short reference recording. Voice cloning requires accurate content and preserved speaker identity, but supervised acoustic-token prediction does not directly optimize these waveform-level properties. Reward-based post-training addresses this mismatch, but in discrete diffusion, token choices and reveal positions jointly define the sampling trajectory, complicating alignment. We introduce RAWD-TTS (Ratio-free Advantage-Weighted Denoising), which scores decoded samples with recognition and speaker rewards and uses group-relative advantages to weight masked-token reconstruction of those samples, without reverse-trajectory likelihoods or target audio. On 500 Russian CV3-Eval voice-cloning prompts, joint alignment reduces word error rate from 3.18% to 2.42% at the reward-selected checkpoint (24.0% relative) and to 2.58% at the final checkpoint (19.0%), while WavLM speaker cosine rises from 0.733 to 0.748 and 0.755. Controlled experiments characterize recognition-identity trade-offs and the effects of corruption count, group composition, and weighting.