ParA-LLM improves understanding of speaker and acoustic speech traits
ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding
SoundArtificial IntelligenceMachine Learning
Summary
Current speech AI systems can recognize words but often miss important details like who is speaking, how they feel, or the sound environment. The authors created a new system called ParA-LLM that learns to identify 22 different speech features such as speaker traits and background sounds by training on many question-answer pairs. Their system performs better than existing models at recognizing these extra speech details. They also provide a new benchmark to test improvements in this area.
What this means in practice
- •For voice assistant developers: Improve voice assistants’ ability to detect speaker emotions and background noise for more natural interactions.
- •For call center technology teams: Enhance customer support systems by recognizing speaker characteristics and acoustic environments to better assess call quality and sentiment.
Authors
Nishit Anand, Jiaqi Su, Ke Chen, Yunyun Wang, Dinesh Manocha, Ramani Duraiswami, Rithesh Kumar, Zeyu Jin
Abstract
Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.