Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings
2026-06-08 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageMachine Learning
AI summaryⓘ
The authors studied how ranking AI models by having people compare their answers works in practice. They found that these pairwise comparisons, when combined using a method called Elo, closely match rankings based on actual correct answers. Even when judges are not very good or have biases, the rankings stay reliable. The authors also discovered that judges tend to prefer answers that repeat the final response, which influences their choices a bit.
pairwise comparisonsElo rating systemgenerative modelsSpearman correlationjudge biasbenchmark evaluationaccuracy rankingstyle biascausal driver
Authors
Mina Remeli, Moritz Hardt
Abstract
Pairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they reward superficial stylistic cues or display judge biases. In a more positive turn, we show that model rankings from pairwise comparisons strongly agree with ground-truth-based accuracy rankings when such ground truth is available for comparison. By converting five well-known benchmarks into free-form generative evaluations, we find that Elo rankings achieve a Spearman correlation above 0.9 with accuracy rankings and substantially outperform direct evaluation when the judge is weak. Furthermore, style and judge bias have only minor effects on model rankings, despite most judgments occurring on pairs where both candidate answers are correct (or incorrect). On such pairs, we find that repetition after the final answer (echo) is a causal driver of judge preference.