Papers for

broadcast captioning teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Bilingual speech system recognizes words and vocal sounds together

Source-Adaptive Data Curation for Bilingual NVV-Aware ASR

Abstract: Nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, convey affective and interactional information that conventional automatic speech recognition (ASR) systems often discard. We present a bilingual Mandarin-English system for Track 1 of the NVVSpeech Challenge at ISCSLP 2026, which requires joint transcription of lexical content and 16 NVV categories at their transcript-relative positions. Our NVV-Aware Whisper adapts Whisper-medium through checkpoint-compatible vocabulary remapping, enabling lexical tokens and inline NVV tags to be decoded within a unified autoregressive sequence without expanding the vocabulary. To provide reliable and diverse supervision, we further introduce a source-adaptive data curation strategy that refines public NVV corpora through acoustic augmentation and multimodal LLM filtering, while mining spontaneous NVVs from in-the-wild media through automated preprocessing and annotation. Under the official bilingual evaluation protocol, the proposed system improves final score from 33.32 to 53.61, with ablations confirming the complementary benefits of the proposed data-curation components.

Wed 9 SeptSound
The gist
Many speech recognition systems ignore sounds like laughter or coughs that people make while talking. The authors created a program that listens to both Mandarin and English speech and also detects 16 types of these nonverbal sounds in the exact spots they happen. They improved an existing speech model so it could handle these extra sounds without making the system more complex. To train it well, they gathered and cleaned up lots of real-world speech data that included these sounds. Their final system works much better than before in recognizing speech and nonverbal sounds together.
Open 2609.09929v1