VoTSpeech separates voice design from speech generation in TTS systems
VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS
SoundComputation and Language
Summary
Making computer voices that follow instructions is hard because voice features are mixed up with the way speech sounds are made. The authors created VoTSpeech, which first turns text instructions into a clear voice representation before turning that into speech sounds. This extra step helps the system better control how the voice sounds while speaking. Tests show VoTSpeech matches or beats other open systems and sounds natural.
What this means in practice
- •For voice assistant developers: Create voice assistants that customize speech style directly from text instructions for more natural and controllable spoken responses.$Commercial implications: Enables development of customizable AI voices for commercial products like virtual assistants and automated customer service agents.
- •For multimedia content creators: Generate spoken content with specific voice characteristics from simple text prompts, improving voice variety without extra audio recordings.
Authors
Wenbing Yang, Qihang Lu, Zihan Sun, Peilei Jia, Yingming Gao, Ya Li, Jun Gao
Abstract
Instruction-following text-to-speech (TTS) typically leaves voice design implicit, as conventional speaker embeddings encode reference audio rather than text-driven targets. We propose VoTSpeech, which decouples voice design from speech generation via an explicit intermediate voice latent. Built on a shared language model backbone, VoTSpeech first predicts a continuous voice latent from text instructions using a flow-based module. This latent serves as a shared condition for both the language model and the Acoustic DiT, guiding autoregressive sequence modeling and acoustic latent generation, respectively, before an audio VAE decoder converts the acoustic latents into waveforms. To capture speaker identity and complementary acoustic nuances, the voice latent is jointly supervised by speaker embeddings and temporally pooled audio VAE features. With task-specific adaptation, VoTSpeech performs strongly on InstructTTSEval-ZH, demonstrating competitive capabilities against strong open-source models while maintaining high naturalness. These results confirm that explicit voice latent design and its conditioning interface effectively empower text-driven voice design for controllable speech synthesis. Code, model, and audio samples are available at https://github.com/ywbn/VoTSpeech .