The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel
2026-06-22 • Computers and Society
Computers and SocietyArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors point out that many tools used to measure political views work well in Western countries but fail elsewhere. They propose a new method that treats a large language model like just one judge among many, combining multiple opinions for better results. Their tests show this method is reliable and that disagreements often arise from different interpretations, not mistakes. They demonstrate this approach using data from the Middle East and North Africa but believe it can apply to other regions too. They also acknowledge the method still needs human validation.
political positionsmanifesto codingexpert surveystext-scaling modelslarge language modelKrippendorff's alphainter-rater reliabilityvalidityMiddle East and North Africainterpretation
Authors
Tarek Gara
Abstract
Most tools for measuring political positions, manifesto coding, expert surveys, text-scaling models, were built and validated on Western party systems, and outside that setting they work poorly, and often not at all. This paper is an attempt at a method for those settings. It treats a large language model not as a measurement device but as a single, fallible rater in a panel, roughly the way an expert survey treats one expert: the value comes from pooling many judges rather than trusting any one of them. I describe the panel, an applicability rule that keeps a score of zero distinct from a blank, and a lens system that separates what an actor says from what it does. I report three results. First, holding a definition-free round fixed, adding written axis definitions moves scores by a mean of 1.8 points on a 21-point scale and tightens agreement between raters (mean absolute gap 2.81 to 2.50; r 0.81 to 0.89); they make two independent raters agree more closely, which an arbitrary steer would not. Second, across nine models from eight laboratories in two countries, Krippendorff's alpha is 0.86 on both an interval and an ordinal metric, and it stayed put as the panel grew from five raters to nine. That is reliability, the reproducibility of a reading, and not validity, its correctness. Third, where the panel does disagree, the disagreement is informative: the sharpest split, a full-scale divergence on an actor's stance toward its state's foundational order, points to a referent problem, and a blind triple-coding puts about two-thirds of it down to interpretation rather than error. I try to be plain about what the method can't do, including the human validation it still lacks, and I release the instrument and data in full. The worked example is the Middle East and North Africa, but I'd expect the method to carry to any region these standard tools leave out.