Large language models improve emotion detection from speech and text
Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue
Computation and LanguageArtificial IntelligenceMachine LearningMultimedia
Summary
Understanding emotions in conversations is hard, especially when people talk and express feelings in many ways like voice tone and words. The authors show how big language models can rate emotions along several scales like positive or negative feelings, energy level, and control in conversations that include both speech and text. They find that tuning smaller language models specifically for this task works better than just using larger general models, especially when adding descriptions of how speech sounds. Their best model sets a new accuracy record on a common test dataset of emotional conversations.
What this means in practice
- •For customer service teams: Improve emotion detection in calls by combining voice tone descriptions with text using fine-tuned language models for better customer satisfaction insights.
- •For game developers: Use language model-based emotion scoring to create more emotionally responsive characters reacting to player speech and dialogue.
Tested on one dataset.
Authors
Yutong Hu, Jinho Choi
Abstract
Emotion recognition in conversation has been widely studied, but applying Large Language Models (LLMs) to continuous dimensional emotion evaluation in multimodal dialogue remains largely unexplored. We propose an LLM-based framework that performs discrete emotion recognition and Valence-Arousal-Dominance (VAD) dimensional evaluation on IEMOCAP, incorporating acoustic cues as natural language descriptions following the SpeechCueLLM approach. We evaluate six models spanning the LLaMA, GPT, and Qwen families under zero-shot prompting, few-shot prompting, and LoRA fine-tuning. LoRA fine-tuned LLaMA models substantially outperform prompt-engineered GPT models on both tasks despite GPT's larger scale, a gap we attribute to domain adaptation rather than model capacity. Our best model achieves a Valence CCC of 0.7822, a new state-of-the-art on IEMOCAP. Ablation studies confirm that textual audio descriptions meaningfully improve smaller models (+3.5 to 3.6 weighted F1) while contributing little for the largest model, suggesting audio cues are most valuable when linguistic capacity is limited. The performance asymmetry across VAD dimensions closely mirrors the annotator agreement hierarchy in IEMOCAP's own annotations.