Emotion classifiers disagree on fear and surprise across datasets
Fear Is Not Always Fear: Variation in Emotion Label Equivalence Across Datasets
Computation and Language
Summary
Emotion detection tools often use labels like fear or surprise assuming they mean the same thing everywhere. The authors show that different datasets interpret these emotion labels quite differently. For example, fear classifiers trained on separate datasets sometimes classify the same text very differently, even if they roughly agree on the order of fear intensity. Also, when models are applied outside their original data, they lose accuracy and can give conflicting answers about which texts show emotions like surprise. This means an emotion label name alone doesn’t guarantee consistent measurement across sources.
What this means in practice
- •For natural language processing teams: Choose emotion training data carefully since different datasets produce conflicting emotion detection results, impacting system reliability.
- •For social media monitoring teams: Be cautious interpreting surprise or fear metrics across tools trained on different emotion datasets, as results may contradict.
Authors
Emily Öhman, Anna Koufakou
Abstract
Emotion classifiers are routinely applied beyond their training data, on the assumption that same-named labels such as fear or surprise mean the same thing across resources. We test this assumption for five labels shared by GoEmotions, XED, and BRIGHTER, comparing models on identical inputs and under cross-corpus transfer. On the same texts, one fear classifier flags 71% as fearful and another 8%, yet the two largely agree on how the texts rank. Across 30 resource-label comparisons, training-label prevalence explains about half of this kind of scale mismatch, and a fitted intercept removes most of it. What recalibration cannot repair is larger: transferred models lose 55% of native discrimination in aggregate, agreement in ranking between classifiers is only weakly related to that loss, and for surprise the resources disagree about which texts express the emotion. The choice of training resource can therefore change conclusions: on two external tweet collections, BRIGHTER- and XED-trained classifiers give opposite answers about which shows more surprise, even after rank normalization, with intervals excluding zero in both directions. A shared emotion label name does not establish an interchangeable measurement target.