Large language model leaderboards gain reliable anytime error control
Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized e-Processes
Machine Learning
Summary
When comparing language AI models on shared tests, results can vary and cause confusion about which model is truly better. The authors present BB-EDGE, a new way to compare these models that controls the chance of wrongly declaring a model superior, no matter when or how often the leaderboard is updated. BB-EDGE treats comparisons as a network of pairwise tests and uses mathematical methods to ensure the reliability of rankings and top model lists at any point in time. Tests on simulated and real data show this method keeps errors low while efficiently certifying model rankings.
What this means in practice
- •For machine learning platform teams: Provide model evaluation dashboards that give statistically reliable rankings updated continuously without risk of false superiority claims.
- •For statistical software developers: Implement anytime-valid error control in software that supports benchmarking and ranking of competing algorithms.
Authors
Hongfu Gao, Songxin Zhang, Zejian Xie, Bingyi Jing, Zhou Wang, Yiming Liu
Abstract
Large language model (LLM) leaderboards compare model capabilities by ranking models according to their mean performance on fixed benchmarks. However, variability in evaluation outcomes across runs may produce unsupported claims of model superiority on the benchmark, a risk compounded by leaderboard updates. In this paper, we propose BB-EDGE (Benchmark-Weighted and Block-Factorized e-processes for Directed Graph Evaluation), a principled framework that represents an LLM leaderboard as a directed graph whose edges certify pairwise mean-performance advantages, with anytime-valid family-wise error rate (FWER) control. Concretely, for each direction, BB-EDGE constructs an empirical-Bernstein e-process by factorizing evidence over protocol-defined blocks and assigning stakes proportional to the corresponding block weights, then applies direct e-Holm across these $e$-processes to certify directional advantages as edges. Theoretically, we characterize weight-proportional linear stakes under heterogeneous benchmark-average nulls and prove anytime FWER control under arbitrary within-block and cross-pair dependence. BB-EDGE further supports anytime-valid Top-$k$ certification and simultaneous rank intervals. Extensive experiments on synthetic data and four real-world benchmarks demonstrate that BB-EDGE maintains anytime FWER control while achieving high efficiency.