Audio language models improve emotion detection in TV speech noise
TVPara: Paralinguistic Captioning of Expressive TV-Series Speech in Complex Acoustic Environments
Sound
Summary
It is hard for large audio-language models to understand emotions and other speech feelings when voices are mixed with noisy backgrounds. The authors created a new dataset called TVPara using speech from TV shows, which includes real emotions and background sounds found in those scenes. They trained models on this data and saw a big improvement in the models’ ability to recognize emotion in tricky sound environments. This improvement also worked on new TV shows and cleaner, different types of speech.
What this means in practice
- •For speech technology developers: Enhance voice assistants and speech recognition systems to better interpret emotions and speech nuances in noisy real-world settings using TVPara-trained models.
- •For media analytics teams: Analyze emotional content and speaker expression in TV show dialogue even when background noise or music is present by using models fine-tuned on TVPara.
Authors
Ziyang Zhang, Wenyi Yu, Wen Wu, Xuenan Xu, Chao Zhang
Abstract
Large audio-language models (LALMs) struggle to parse paralinguistics (e.g., emotion, pace, pitch) in speech that combines intense emotion with complex acoustic environments. Strengthening this ability calls for training data that reflects such conditions. However, existing corpora rely heavily on studio-recorded speech, offering limited emotional and acoustic diversity. To address this gap, we introduce TVPara, a paralinguistic captioning dataset built from TV series. Speech in TV series often carries diverse emotional expression and co-occurs with scene-specific background sound, providing expressive emotion and complex acoustic environments at scale. Constructed via LALM pre-annotation and human correction, TVPara includes two tasks: Single-Utterance (~492h) and Multi-Utterance (~213h) Paralinguistic Captioning. Fine-tuning open-source LALMs on TVPara yields an average relative improvement of 31.26% in paralinguistic understanding under complex acoustic environments. The gains further generalize to an unseen TV series and to cleaner, cross-domain speech.