LLM leaderboard claims are often unsupported by hidden model choices
How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?
Machine Learning
Summary
Leaderboards that rank large language models (LLMs) show which models perform best, but many claims rely on selecting from many hidden versions not made public. The authors study how many hidden model attempts can explain a reported improvement without proving a real advantage. They find that most close leaderboard differences lack strong statistical support and that certifying a win depends on assumptions about how similar the hidden versions are. Their method makes these assumptions clear without needing to know all the hidden models tried.
What this means in practice
- •For machine learning engineers: Check whether claimed LLM improvements on leaderboards truly indicate better models or just result from hidden trial selections.
- •For ai benchmarking teams: Design evaluation protocols that account for hidden model choices to reduce misleading leaderboard rankings.
Authors
Chen Yang, Xianyang Zhang, Jun Chen
Abstract
LLM leaderboard gains can reflect selection among privately evaluated model variants, yet neither the number of variants nor their dependence is public. We ask how many hidden variants a published margin can support while retaining statistical evidence of a provider's advantage over a fixed comparator. For a fixed candidate family under a Gaussian margin model, we derive a sensitivity curve that reports this maximum count as a function of a lower bound on within-family correlation. The relevant correlation must match the score used for ranking and the sampling model: in a controlled family, pooled item correlation is 0.90, whereas composite-score correlation is 0.46 under item resampling and 0.92 when MMLU subjects are resampled. An item-based audit of 394 adjacent-rank claims on the Open LLM Leaderboard finds that 391 lack statistical support even before accounting for selection. Among claims that pass the uncorrected test, certification can depend on assumptions about the hidden family's correlation. The resulting curves make these assumptions explicit without estimating the unobserved search size.