DiTAR plus improves speech synthesis for long complex utterances
DiTAR+: Dual Optimization for Robust Autoregressive Diffusion Speech Synthesis
SoundArtificial Intelligence
Summary
Generating speech with computers can be tricky when sentences are long or complicated, leading to mistakes in pronunciation or meaning. The authors found that existing speech models sometimes lose track of important information over time or get confused by sound details. They created a new framework called DiTAR+ that looks at longer context and separates understanding of meaning from sound details. This makes the computer-generated speech clearer, more accurate, and better at sounding like the original speaker, especially for challenging sentences.
What this means in practice
- •For speech technology developers: Improve stability and accuracy of synthesized speech for long or complex utterances in text-to-speech systems.
- •For voice assistant engineers: Enhance robustness of voice generation when responding with lengthy or complicated spoken replies.
- •For audiobook producers: Create synthetic voices that maintain speaker identity and reduce errors over long narration segments.$Commercial implications: Enables commercial-quality synthetic audiobooks by significantly reducing pronunciation errors and preserving consistent speaker voice.
Authors
Ziyu Zhang, Tianlun Zuo, Hanzhao Li, Haoyu Zhang, Lei Xie
Abstract
Continuous-latent Autoregressive Diffusion Transformer (AR-DiT) models have demonstrated immense potential in zero-shot speech generation. However, they still suffer from limited decoding stability when synthesizing long utterances or complex linguistic structures. This instability primarily stems from a restricted historical receptive field and an acoustic inertia dependency within the diffusion decoder, which causes the model to ignore semantic conditions. To address these challenges, we propose DiTAR+, a dual-optimization framework. First, we introduce Dilated Context Sampling to expand the macro-level historical receptive field without violating physical temporal continuity, thereby preventing cumulative error propagation. Second, we propose Hierarchical Acoustic Masking to prevent shallow layers from attending to acoustic pre-context, explicitly decoupling semantic alignment from acoustic detail reconstruction. Extensive experiments show that our framework effectively mitigates pronunciation errors and semantic hallucinations, enhances generation robustness on challenging sentences, and maintains exceptionally high speaker similarity throughout the entirety of long-form utterances. On the linguistically challenging ZH-Hard set, DiTAR+ reduces the word error rate from 12.478% to 9.893%, and on extended utterances of 25 to 35 seconds it improves speaker similarity from 0.741 to 0.759 while simultaneously lowering the word error rate from 2.778% to 2.173%, outperforming both discrete-token and pure flow-matching baselines.