Instability in fixed large language model judges affects ranking precision
Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards
Computation and Language
Summary
People often assume large language models (LLMs) used to judge other AI models give consistent verdicts when set to a fixed state. This paper finds that even when using the same model snapshot and settings, LLM judges produce different answers about 5% of the time, and much more on difficult cases. These fluctuations make it hard to tell small differences between models on leaderboards, creating a noise floor that limits confidence in rankings. The authors suggest reporting stability measures and using multiple judges to better understand how reliable evaluation results are.
What this means in practice
- •For machine learning engineers: Improve AI model evaluation by incorporating measures of judge verdict stability and multiple judge perspectives to avoid misleading leaderboard rankings.
- •For ai product managers: Use stability profiles and judge re-runs to make more informed decisions about model selection and deployment based on reliable evaluation results.
Authors
Krishna Chytanya Ayyagari
Abstract
Modern LLM evaluation assumes that pinning a judge to a fixed model snapshot and decoding at temperature zero yields reproducible verdicts. We show this assumption fails as a property of how LLM-as-Judge is operationalized on cloud serving infrastructure, not of any particular model family. Across four frontier judges all served via a single major enterprise cloud platform and three standard benchmarks (Arena-Hard, AlpacaEval 2, MT-Bench), identical inputs to the same pinned, temperature-zero judge produce different verdicts across re-runs: per-item flip rates of roughly 5% on average and about 40% on the close-call items that decide leaderboard margins, with a per-judge magnitude spanning a 40x range (from 0.13% to nearly 10%). We introduce metrics tailored to this instability: per-item flip rate, a two-part stability profile (waver fraction and conditional intensity), and adjacency separability - and report what the variance does and does not do to rankings. For a single judge the aggregate ranking is stable (0% top-K instability, 0% pooled winner flip); what degrades is precision: under a paired hierarchical bootstrap, roughly one-fifth to three-quarters of adjacent leaderboard positions are statistically indistinguishable, a noise floor we attribute primarily to finite prompt sampling rather than to the judge. Across judges, leaderboards agree on the coarse ordering but diverge in the middle (Kendall's tau as low as 0.42-0.64 between families on Arena-Hard), and of 13 published head-to-head ranking claims we re-judge, 5 fail under a defensible judge swap or re-run. We argue that leaderboards report unhedged point estimates that misrepresent the noise floor of the instrument, and we propose a minimal, low-cost reporting protocol: several judge re-runs, published stability profiles and adjacency intervals, and results under at least two judges from different families.