Text to speech model improves natural sound and streams fast on a single GPU
TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context
SoundComputation and LanguageMachine Learning
Summary
Text-to-speech systems often struggle to sound natural while running quickly and on normal computers. The researchers propose TontaubeV1, which uses a special way to break down speech into meaningful parts and sound details separately. This helps the system predict the meaning and timing first, then add finer sounds without delay. Their approach can handle streaming speech in real time using just one common graphics card, and it works well in English, German, and some other languages. Tests show it sounds as good or better than several existing popular speech models.
text-to-speechprosodystreaming inferencehierarchical codectransformer modellatent spacevoice conditioningreal-time factormultilingual speech synthesis
Authors
Fritz Cremer, Jonathan Cremer
Abstract
Text-to-speech systems often face a trade-off between natural prosody and efficient inference: higher perceptual quality typically comes at increased computational cost and latency. We present TontaubeV1, a model that preserves natural prosody while enabling streaming from a single consumer GPU. Speech is encoded by the hierarchical DualCodec representation at 12.5 Hz, which separates a semantic stream from successive acoustic refinements. Our design assumes that prosodic structure is largely established when the semantic stream is generated, and allocates capacity accordingly: a Qwen3-1.7B-derived transformer predicts that stream and thereby the utterance duration, while three progressively smaller Qwen3-0.6B-derived transformers each add one acoustic refinement. Text is tokenized per character rather than by subword. Paired text and audio markers at shared positions support long-form generation with bounded context, and overlapping DualCodec reconstructions are mapped into the VibeVoice acoustic latent space and decoded causally, enabling streaming despite DualCodec's noncausal decoder. The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German, with additional multilingual support. The four predictors total 2.9B parameters; on a single RTX 5090 the streaming path reaches approximately 200 ms to first audio. In separate non-streaming measurements, the end-to-end real-time factor (RTF) is 0.08 for one input and the aggregate RTF is 0.02 across eight concurrent inputs. On our LLM-as-a-judge audiobook-reading benchmark, TontaubeV1 matches ElevenLabs Flash v2.5 and outperforms Fish Audio S2 Pro, the April 2026 Gradium API, and Cartesia Sonic 3 on prosody. The model weights are released on Hugging Face under the Tontaube Community Model License 1.0.