Expressive zero-shot text to speech with controllable style and timbre
E3-System: Expressive EmoData, Explicitly Controllable, Easy-to-Use TTS
Sound
Summary
Text-to-speech systems can mimic new voices but often struggle to express emotions or styles clearly. The authors built E3-System, which lets users control how speech sounds by typing style descriptions or providing example voices. It keeps the speaker’s unique voice while adding expressive emotions or styles. They created a large speech dataset and used clever techniques to separate voice identity from style, making expressive speech easy to generate for many voices.
What this means in practice
- •For voice assistant developers: Generate expressive synthetic speech in different emotional styles for any user voice using simple text or speech descriptions as input.
- •For audiobook producers: Produce narrations with natural emotional variety by controlling speech style separately from narrator voice.
Authors
Yiqiao Huang, Xiaolong Lin, Zhenpeng Zhan, Zhizheng Wu
Abstract
Recent zero-shot Text-to-Speech (TTS) systems can clone unseen speakers, yet controllable expressiveness remains limited by a gap between semantic intent and acoustic realization. Text descriptions offer a natural control interface but provide weak acoustic grounding, whereas speech style prompts provide concrete prosody but entangle style with the prompt speaker's timbre. To bridge this gap, we propose E3-System, a unified intent-to-speech framework for expressive zero-shot TTS that couples three requirements: Expressive style modeling, Explicit timbre-style control, and Easy-to-use natural-language interaction. Given a target speaker and either a natural-language style description or a speech style prompt, E3-System generates expressive speech while preserving the target timbre. The key idea is to close the full control loop: building a large expressive style space, grounding user intent into usable acoustic style references, and synthesizing speech with disentangled timbre and style. To realize this loop, EmoData provides a thousand-hour multilingual emotional speech dataset, Style-RAG grounds text descriptions into high-quality acoustic style prompts from these data, and Dual-TTS transfers the grounded style to arbitrary target speakers through VC-augmented timbre-style disentanglement. Experiments show that E3-System achieves strong speech quality, speaker preservation, and explicit expressive controllability.