Compact text to speech model improves naturalness and speaker similarity
Training a Compact, Strong, and Realistic Text-to-Speech Model
SoundArtificial Intelligence
Summary
Modern text to speech systems often produce voices that are clear and match the speaker, but they lack the natural flow and expression humans have. The authors created a smaller model named Audio8-TTS that can copy voices without extra training and sounds more natural. They trained it using a special system that judges how realistic the speech sounds, not just how close it is to the original audio. This approach helped improve how natural and expressive the synthetic speech feels while keeping it easy to understand.
What this means in practice
- •For voice assistant developers: Create more natural and expressive synthetic voices that better match target speakers for virtual assistants.
- •For audiobook producers: Generate high-quality synthetic narrations that maintain speaker identity and sound more like real human performances.
Authors
Runyuan Cai, Yu Lin, Yiming Wang, Xiaodong Zeng
Abstract
Text-to-speech (TTS) systems are conventionally optimized for intelligibility and speaker similarity. However, neither objective captures the prosodic detail that distinguishes human performances from synthetic speech, leaving a realism gap that larger models and higher-quality data can narrow but not close. In the absence of an objective that directly targets this gap, realism remains a persistent challenge in TTS. We present Audio8-TTS, a compact 0.6B-parameter model for zero-shot voice cloning that achieves state-of-the-art performance in intelligibility and speaker similarity, with particularly strong results in perceptual naturalness. To align the training signal with human perception rather than signal-level fidelity, we construct a quality ladder from widely used open-source TTS systems and train a realism critic to assess generated speech against matched human recordings of the same sentences. An identity-erasure objective discourages the critic from relying on generator fingerprints. We train the model in three stages: multilingual supervised fine-tuning; group relative policy optimization (GRPO) with word error rate (WER) and speaker similarity (SIM) rewards; and GRPO guided by the realism critic. The final stage improves perceived prosodic naturalness and speaker similarity while preserving WER performance.