NVV-Locator detects laughter sighs and coughs precisely in speech
NVV-Locator: From Transcript Tags to Acoustic Boundaries for Fine-Grained Nonverbal Vocalization Grounding
Sound
Summary
People often add sounds like laughter, sighs, and coughs when they talk, and these sounds help express feelings or reactions. The authors developed a system called NVV-Locator that finds exactly when these sounds happen in speech recordings. They combined information from many sources to train their system using precise timing data and created a benchmark set to test it. NVV-Locator outperforms current audio models at detecting these nonverbal sounds and works well even on data it wasn't trained on.
What this means in practice
- •For voice assistant developers: Improve voice assistants by precisely identifying user nonverbal sounds like laughter or sighs to better understand emotional context during interactions.
- •For call center technology teams: Automatically detect subtle nonverbal cues such as coughs or breaths in customer calls to enhance sentiment analysis and agent training.
Authors
Yuang Cao, Bingshen Mu, Zhennan Lin, Guojian Li, Haoyue Zhan, Jie Liu, Chuan Xie, Qiang Zhang, Liumeng Xue, Lei Xie
Abstract
Human speech includes nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, which convey affective and interactional information. Existing approaches typically represent NVVs as transcript-level tags, providing limited supervision for their waveform-time boundaries. We present NVV-Locator for fine-grained NVV temporal grounding. We first unify 26 NVV categories across public resources and construct large-scale timestamp-supervised training data through dual-LLM verification, transcript-guided forced alignment, and energy-based boundary refinement. We further introduce NVV-TimeBench, an expert-refined benchmark with 667 utterances and 1,094 events. NVV-Locator uses a non-autoregressive slot-filling architecture to jointly predict lexical timestamps, NVV categories, and event boundaries. On NVV-TimeBench, it achieves 71.0% Micro F1, 70.2% Macro F1, 80.4% Macro mIoU, and 59.6 ms Macro mMAE, outperforming the evaluated large audio model counterparts. Evaluation on an external corpus further demonstrates the cross-corpus generalization of NVV-Locator.