Speech system improves natural sounds with better nv vocalizations
Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations
Artificial Intelligence
Summary
Making speech sound natural and expressive can be tricky, especially when including nonverbal sounds like laughs or sighs, which are very different and often rare in speech recordings. The authors created a system that treats these nonverbal sounds as special tokens during training and uses smart data tricks to balance how often each sound appears. Their system learns from lots of data in both Mandarin and English, improving its ability to generate these sounds correctly. Tests show it performs very well, ranking top among similar systems.
What this means in practice
- •For voice assistant developers: Improve voice assistants by generating more natural nonverbal sounds like laughs or sighs to enhance user interaction.$Commercial implications: Enables commercial voice products with expressive speech, increasing user engagement through better emotional realism.
- •For game audio engineers: Create more expressive character speech by controlling nonverbal vocal sounds to add realism and immersion in games.
Authors
Ziyu Zhang, Yun Chen, Taihui Wang, Hanzhao Li, Qicong Xie, Rilin Chen, Zhixian Zhao, Lei Xie
Abstract
Controllable synthesis of nonverbal vocalizations (NVVs) is es- sential for natural and expressive speech, but remains challeng- ing due to their acoustic diversity and imbalanced distribution in existing corpora. To address these challenges, we develop an NVV-aware DiTAR system that models continuous speech latents, encodes the 16 target NVV categories as dedicated to- kens, and adapts stop prediction to distinguish mid-utterance vocalizations from utterance boundaries. Training begins with large-scale bilingual pre-training on diverse NVV speech, fol- lowed by continued supervised fine-tuning on a corpus en- hanced through targeted synthetic augmentation and frequency- aware rebalancing. At inference time, we select the acoustic prompt, tune the LM-guidance and noise-injection scales, and apply Best-of-N sampling with multi-metric selection to re- duce generation failures. The final system achieves an official weighted bilingual score of 62.786, ranking first in Mandarin, second in English, and first overall among participating systems in Track 2 of the ISCSLP 2026 NVVSpeech Challenge. Ab- lation studies show that targeted augmentation benefits under- represented NVV categories the most, while robust candidate selection requires balancing NVV correctness, lexical fidelity, and perceptual quality.