Emotion labels often mix positive and negative feelings in text
Mixed Feelings, Single Polarity
Computation and Language
Summary
Many emotion datasets try to label feelings as simply positive or negative by giving each emotion a single 'polarity.' The authors found this approach works well only when the text's emotions are clearly in one direction. But for texts expressing both positive and negative emotions at once, this method often fails and leads to errors. They show that such mixed emotions are common in real annotations and suggest handling these cases differently, such as keeping a 'mixed' label instead of forcing a single polarity.
What this means in practice
- •For sentiment analysis teams: Improve emotion detection in social media by identifying and correctly labeling mixed or conflicting emotions rather than forcing a single sentiment polarity.
- •For customer support ai developers: Enhance chatbot or support systems by better recognizing when user messages express both positive and negative feelings to tailor responses more sensitively.
Authors
Emily Öhman, Saif Mohammad
Abstract
Emotion datasets are routinely mapped onto positive and negative valence, by assigning each emotion category one polarity. We test where this works and where it fails, using SemEval-2018 Affect in Tweets (English, Arabic, Spanish), Affective Text aligned with EmoBank, raw GoEmotions annotations, naturalistic hashtag corpora, and a controlled annotation study. Fixed projection works well when a text carries labels of one valence. It fails on a subset identifiable from the labels alone: tweets whose emotion labels point in opposite directions. These are 9.0% of polar SemEval test tweets but cause 54.8% of the polarity errors of a learned projection, are more than twice as likely to receive a neutral-or-mixed gold label, and are not rescued by modeling the full label set. Annotators produce such configurations routinely: 16.3% of multilabel GoEmotions judgments combine emotions from the dataset's own positive and negative parents. In a controlled study holding word and reader constant, context changes the coarse valence judgment in 56.8% of cases, against 19.9% for controls. We recommend projecting label configurations rather than labels, abstaining or keeping a mixed class where labels conflict, and reporting coverage alongside accuracy.