Resolution Diagnostics for Paired LLM Evaluation
2026-05-28 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors studied how large language models (LLMs) are compared on leaderboards and found that many pairwise rankings between models are not statistically clear or resolved. They treated the comparison as a hypothesis test and introduced a metric called the resolution ratio to measure how well pairs can be distinguished. They also showed that a common shortcut method underestimates the sample size needed to reliably detect small differences between models. This uncertainty in rankings remains even after correcting for multiple tests and using more complex testing methods.
large language modelspairwise comparisonhypothesis testingstatistical powersample size estimationCohen's dmultiplicity correctionsequential testingleaderboardsbootstrap resampling
Authors
Anany Kotawala
Abstract
Across two public LLM leaderboards, many displayed pairwise rankings do not meet a conventional paired-test resolution target under the actual paired evaluation design: 11 of 40 Open LLM Leaderboard v1 pairwise comparisons and 4 of 9 MMLU-Pro top-10 adjacent-rank pairs are unresolved at (alpha, 1-beta) = (0.05, 0.8). The MMLU-Pro count rises to 6/9 under real subject-level clustering and stays at 5-6 out of 9 in 99.9% of category-bootstrap resamples. We frame paired LLM evaluation as a hypothesis-testing problem, invert level-alpha, power-(1-beta) tests, and report a per-pair resolution ratio q = N/N* as the primary diagnostic. A sharp small-effect expansion with an explicit second-order constant shows that the widely-used unpaired Cohen-h-plus-(1-rho) shortcut deviates from the correct N* by approximately a factor of two in the close-comparison regime, a deficit that three of five off-the-shelf calculators(Cohen 1988, G*Power, R pwr) silently inherit when the user post-multiplies their per-arm output by (1-rho). The unresolved-pair pattern remains under multiplicity correction and anytime-valid sequential testing.