End-to-End Training for Discrete Token LLM based TTS System
2026-06-08 • Sound
SoundArtificial Intelligence
AI summaryⓘ
The authors improved text-to-speech (TTS) by training all parts of their system together at once, instead of separately. They combined the speech tokenizer, language model, and sound generation model into one end-to-end framework. This helped the system better understand and produce speech sounds accurately and naturally. Their approach achieved state-of-the-art results on a speech benchmark with lower error rates than previous methods. This shows that training the whole system jointly can make TTS systems simpler and more effective.
text-to-speech (TTS)speech tokenizerlarge language model (LLM)diffusion modelflow-matchingend-to-end trainingreward modelword error rate (WER)multi-task learningSeed-TTS-Eval benchmark
Authors
Changfeng Gao, Yong Ren, Jun Yuan, Ye Bai, Zhao You, ShiDong Shang
Abstract
Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently. In this paper, we propose a fully end-to-end (E2E) optimization framework that unifies the training of the speech tokenizer, LLM, FM model, and an additional reward model (RM). Specifically, we first jointly optimize the tokenizer using multi-task objectives derived from reconstruction for FM, next-token prediction for LLM, and multi recognition task for RM. This joint training encourages the discrete speech token space to capture acoustically and semantically salient information that is better tailored to TTS. We then further optimize the LLM using downstream reconstruction and recognition by FM and RM, which reduces inference-time mismatch and steers the LLM toward more preferred generations. Experimental results show that our E2E framework consistently outperforms cascaded baselines. On the Seed-TTS-Eval benchmark, our system achieves a word error rate (WER) of 0.78% and 1.56%, a new SOTA result with a 0.6B-parameter LLM and 0.5B-parameter FM model. These results validate that holistic E2E optimization is critical for improving discrete-token-based TTS systems with a much simpler training pipeline.