Selective model use improves sentiment detection for news entities
Finding Errors Is Not Enough: Selective Model Escalation for Entity-Targeted Sentiment Analysis
Computation and Language
Summary
When computers try to figure out if something in the news is said positively or negatively, just finding mistakes isn’t enough to make the system better. The researchers explored smarter ways to fix errors by deciding when to ask a stronger or different model for help. They tested their ideas using news from South Slavic sources and found that sometimes sending tricky cases to a better language model improves results, but it also costs more computing power. Their approach helps balance accuracy and effort when analyzing opinions about specific entities in texts.
What this means in practice
- •For news analytics teams: Improve sentiment detection accuracy by selectively routing uncertain cases to more powerful language models for news monitoring.
- •For customer feedback analysts: Use selective model escalation to better identify sentiments toward specific product entities in limited labeled datasets.
Authors
Nishan Chatterjee, Matthew Purver, Antoine Doucet, Senja Pollak
Abstract
Improving a sentiment classifier with limited resources involves more than simply deciding how much data to collect or which model to train. A labelled collection may lack examples of an important class. Once trained, a classifier may benefit from another model even when that model is weaker on its own. We study these decisions in document-level Entity-Level Sentiment Analysis (ELSA), primarily using South Slavic news. We first compare acquisition orders as collections grow. We then use uncertainty to choose which predictions to revisit, how many to process, and whether an alternative encoder or local LLM can correct them. Finally, we select complete routing policies on development data for held-out evaluation. Minority-prioritised acquisition helps at some very small training sizes, but not consistently thereafter. On SentiCoref, an oracle correcting all errors in the least-confident 10% without harming correct predictions would add 9.14 Macro-F1 points; however the best evaluated action gains only 0.90. Encoder averaging helps on AspectBench and NewsMTSC but harms SentiCoref, where selective LLM assistance is more promising. Both development-selected policies route 30% of records to the encoder-assisted LLM, yielding observed gains of 1.02 points on SentiCoref (interval includes zero) and 1.71 on NewsMTSC, at substantially different inference costs.