Speech systems struggle to recognize and generate nonverbal sounds

The NVVSpeech Challenge 2026: Overview and Lessons for Speech Understanding and Generation with Nonverbal Vocalizations

Sound

Summary

Nonverbal vocalizations like laughter and sighs add meaning to speech, but computers find them hard to recognize and reproduce. The authors looked at ten speech systems tested on Chinese and English sounds, noting issues with breaths and gasps. They found these systems often trade accuracy in word recognition for better detection of nonverbal sounds, and face challenges balancing clear speech generation with natural sounding effects. Human listener feedback helped reveal these strengths and weaknesses. The work highlights what needs improvement to make speech systems better at handling these meaningful sounds.

What this means in practice

  • For voice assistant developers: Improve voice assistants’ ability to detect and produce nonverbal sounds like sighs and laughter to enhance communication naturalness.$Commercial implications: Enables new voice assistant features for more human-like interactions by recognizing and generating meaningful nonverbal sounds.
  • For call center technology teams: Build systems that better interpret nonverbal vocal cues to improve automatic understanding of caller emotions and intents.

A survey. It maps existing work.

Authors

Liumeng Xue, Yuang Cao, Jingbin Hu, Qirui Zhan, Huakang Chen, Yibin Zheng, Yunxiang Chen, Houdun Liu, Su Feng, Haibin Wu, Zheng Lian, Wei Xue, Bengu Wu, Hui Bu, Lei Xie

Abstract

Nonverbal vocalizations (NVVs), such as laughter and sighs, convey meaning alongside words. NVV-aware speech systems must recognize and generate these events while preserving lexical accuracy and perceptual quality. Yet differences in annotation, resources, and evaluation complicate comparisons, and aggregate scores can obscure specific weaknesses. We review ten system descriptions from the NVVSpeech Challenge 2026, covering understanding and generation in Chinese and English across 16 event categories. Our analysis examines supervision construction, task adaptation, and inference-time or preference-based feedback. Results reveal persistent recognition difficulties with breath and gasp and trade-offs between event recognition and lexical accuracy, as well as between generation intelligibility and acoustic quality, with human listening providing complementary evaluation evidence. These findings identify priorities for advancing NVV-aware speech understanding, controllable generation, and comparable evaluation.
View PDFOpen arXiv