Quantifying the Affective Gap: A Zero-Shot Evaluation of LLMs on Fine-Grained Emotion Taxonomies
2026-07-01 • Computation and Language
Computation and LanguageHuman-Computer Interaction
AI summaryⓘ
The authors tested three advanced language models—Claude, ChatGPT, and Gemini—to see how well they can identify 13 different emotions in sentences without any prior examples. They used a consistent test with 1,000 sentences and found that Gemini performed slightly better than the others, but all models struggled with some emotions like love and shame. The differences between the models were not statistically significant, indicating they perform similarly on this task. The study shows that even top AI models have limits in recognizing complex emotions without training on specific examples.
emotion recognitionzero-shot learninglarge language modelsmacro-F1 scoreemotion classificationChatGPTstatistical significanceclass imbalanceaffective computingnatural language processing
Authors
Lawrence Obiuwevwi, Krzysztof J. Rechowicz, Jessica M. Johnson, Vikas Ashok, Sachin Shetty, Sampath Jayarathna
Abstract
Emotion recognition in natural language is a foundational challenge in affective computing, with critical implications for human-computer interaction, mental health support, and conversational AI. This paper presents a rigorous, unified zero-shot evaluation of three leading commercial large language models: Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash). The models were queried through their respective production APIs as of April 2026 on a fine-grained 13-class emotion classification task. Using a stratified 1,000-sentence sample from the boltuix/emotions dataset, which comprises 131,306 sentences across 13 categories, a single uniform prompt with no exemplars was applied identically across all models. Gemini achieves the highest accuracy (39.9%) and macro-F1 score (0.363), followed by GPT-5.4 (38.8%, macro-F1 = 0.291) and Claude (38.0%, macro-F1 = 0.159). All models excel on sarcasm and desire while consistently failing on love, confusion, and shame. McNemar tests reveal no statistically significant pairwise differences (p > 0.10), suggesting convergence at a shared zero-shot ceiling. Claude's markedly lower macro-F1 score exposes a class-imbalance prediction bias. These findings highlight the current limitations of frontier AI systems in zero-shot fine-grained emotion classification.