Sentiment tools and humans struggle to agree on social media feelings
Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts
Computation and Language
Summary
This paper shows that people often disagree when deciding if a social media post sounds positive or negative. The authors compared three popular sentiment tools and three large language models against six people rating 100 tweets. They found that even humans only fairly agreed, and the best computer model still didn’t perfectly match human opinions. The results suggest special training for social media text is important, and humans are still needed to check how well tools perform.
What this means in practice
- •For social media analysts: Choose Twitter-roBERTa-base for more accurate negative sentiment detection when analyzing tweets, supported by human evaluations for better reliability.
- •For marketing teams: Improve campaign feedback analysis by prioritizing binary sentiment judgments and combining model outputs with human input to handle subjective social media posts.
Authors
Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao, Michael L. Nelson, Michele C. Weigle
Abstract
Social media is a rich source of real-time public sentiment, but widely used sentiment analysis tools are often applied without understanding their limitations. In this study, we evaluate the inter-rater reliability of three bespoke sentiment analysis tools (TextBlob, VADER, and Twitter-roBERTa-base) and three large language models (LLMs: Qwen3-32B, GPT-OSS-120B, Llama-4-Maverick-17B) against six human raters across 100 tweets. We measured agreement using two statistical measures: Cohen's kappa for pairwise comparisons and Fleiss' kappa for multiple raters. Even among the human raters, our results showed only fair agreement, highlighting the subjectivity of sentiment analysis. Higher agreement was observed under the binary sentiment classification (negative vs. non-negative and positive vs. non-positive) than under the three-class classification across both humans and automated tools. The Twitter-roBERTa-base model showed the strongest alignment with human ratings, outperforming both bespoke sentiment tools and LLMs, particularly in distinguishing negative versus non-negative sentiment. LLMs showed substantial agreement among themselves and moderate to substantial alignment with humans, performing better in positive vs. non-positive classifications. Our findings underscore that domain-specific fine-tuning remains crucial for reliable social media sentiment analysis, and human-centered evaluation remains essential for establishing gold-standard labels.