Bilingual speech system recognizes words and vocal sounds together
Source-Adaptive Data Curation for Bilingual NVV-Aware ASR
Sound
Summary
Many speech recognition systems ignore sounds like laughter or coughs that people make while talking. The authors created a program that listens to both Mandarin and English speech and also detects 16 types of these nonverbal sounds in the exact spots they happen. They improved an existing speech model so it could handle these extra sounds without making the system more complex. To train it well, they gathered and cleaned up lots of real-world speech data that included these sounds. Their final system works much better than before in recognizing speech and nonverbal sounds together.
What this means in practice
- •For voice assistant developers: Build voice assistants that transcribe speech along with emotional sounds like laughter and sighs for more natural interactions.
- •For broadcast captioning teams: Generate accurate bilingual captions including nonverbal sounds from live or recorded conversations and media broadcasts.
Authors
Yuang Cao, Qirui Zhan, Jingbin Hu, Ziyu Zhang, Yunxiang Chen, Houdun Liu, Shuo Feng, Bengu Wu, Lei Xie, Liumeng Xue
Abstract
Nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, convey affective and interactional information that conventional automatic speech recognition (ASR) systems often discard. We present a bilingual Mandarin-English system for Track 1 of the NVVSpeech Challenge at ISCSLP 2026, which requires joint transcription of lexical content and 16 NVV categories at their transcript-relative positions. Our NVV-Aware Whisper adapts Whisper-medium through checkpoint-compatible vocabulary remapping, enabling lexical tokens and inline NVV tags to be decoded within a unified autoregressive sequence without expanding the vocabulary. To provide reliable and diverse supervision, we further introduce a source-adaptive data curation strategy that refines public NVV corpora through acoustic augmentation and multimodal LLM filtering, while mining spontaneous NVVs from in-the-wild media through automated preprocessing and annotation. Under the official bilingual evaluation protocol, the proposed system improves final score from 33.32 to 53.61, with ablations confirming the complementary benefits of the proposed data-curation components.