Rater agreement model with AI weights improves multi-category scoring reliability
The Rater Ising-Potts Model with LLM-Derived Weights: An Application to Multi-Category Scoring Reliability
Computation and Language
Summary
Measuring how much different people agree when scoring answers can be tricky, especially with multiple score categories. The authors created a new model that looks at how pairs of raters agree and uses AI to understand how similar score categories are. This model doesn’t assume the scores have to be evenly spaced or ordered in a strict way but still keeps the natural order intact. It worked well on different types of test answers, including harder essays, and most mistakes happened between nearby scores, showing the model respects the scoring rules.
What this means in practice
- •For educational test administrators: Assess consistency of human graders on multi-level scoring tasks for better quality control using AI-informed agreement measures.
- •For academic publishing teams: Evaluate reviewer scoring reliability on multi-category paper evaluations by incorporating semantic similarity between categories.
Authors
Matthias von Davier
Abstract
The Ising model is extended to the Potts model for multinomial data. We introduce a Rater Ising-Potts model that uses agreement indicators between pairs of raters and category labels, with weights derived from LLM embeddings. The model does not presuppose ordered category thresholds or equidistant scoring; instead, it focuses directly on pairwise agreement among raters and assigns category-specific positive weights, making it particularly suited for multi-category scoring reliability when raters evaluate responses using a scoring guide. We demonstrate the model's effectiveness on diverse constructed-response tasks, including balanced short-answer items and more challenging, imbalanced essay prompts from the AERA dataset. Across these settings, the model achieves strong agreement with human scores, with the vast majority of misclassifications occurring between adjacent score levels, confirming its ability to preserve the ordinal structure of scoring rubrics without imposing rigid assumptions. A practical similarity normalization and optional power transformation is introduced as a tunable preprocessing step that sharpens semantic distinctions and can be adapted to different datasets. These findings suggest that LLM-derived semantic similarities, combined with this parsimonious Potts-type formulation and flexible similarity scaling, offer a robust and interpretable framework for reliability auditing in educational assessment contexts. Extensions to multiple raters and hierarchical rating processes are discussed.