B-CARE helps spot likely stance classifier mistakes without model knowledge
B-CARE: Blind Classwise Assessment of Relative Evidence for Stance Error Ranking
Computation and Language
Summary
Finding mistakes made by computer programs that decide what people think about a topic can be hard, especially when you only have limited time to check. The researchers created B-CARE, a method that uses a language model to score how well each possible opinion fits some text, without knowing what the original program guessed. By comparing how strong the alternative opinions are against the predicted one, B-CARE ranks which guesses might be wrong. This method works across different programs and datasets without extra training, helping prioritize which answers to double-check first.
What this means in practice
- •For data annotation teams: Prioritize human review of text classification outputs by highlighting likely errors without needing model details.
- •For natural language processing engineers: Integrate a reusable error detection signal that compares stance evidence to improve model monitoring across diverse classifiers.
Authors
Meysam Sabbaghan, Doina Caragea
Abstract
Stance classifiers make errors that can be costly to identify under limited human-review budgets. Prioritizing potentially incorrect predictions remains challenging, particularly across heterogeneous base models. We introduce B-CARE (Blind Classwise Assessment of Relative Evidence for Stance Error Ranking), a prediction-blind framework for identifying high-risk stance predictions. Given only a target and text, a fixed LLM verifier assigns ordinal evidence scores to all candidate stance labels without observing the base model's prediction or confidence. B-CARE constructs a comparative risk score by contrasting the strongest alternative-class evidence with the evidence supporting the predicted class. The resulting classwise evidence profile can be reused across multiple base models without training a separate error detector or calibrating risk scores. We evaluate B-CARE on four stance benchmarks and three heterogeneous base LLMs, covering 12 dataset-model settings and 5,206 unique test instances. Evaluation includes error-ranking discrimination (AUROC and AUPRC), selective prediction (E-AURC), and review effectiveness (ErrorCapture at fixed review budgets). B-CARE achieves a mean AUROC of 0.735 and a mean AUPRC of 0.524, capturing 41.06% of errors within the highest-risk 20% of predictions. Although B-CARE does not uniformly outperform strong confidence-based baselines, it provides a reusable, complementary evidence signal for prioritizing potentially incorrect stance predictions across heterogeneous models.