Ranking models reliably during running benchmarks without errors
Rank Confidence Sequences:Anytime-valid Leaderboards
Artificial Intelligence
Summary
Leaderboards show how well different models perform on a test, but checking ranks repeatedly while tests are still running can lead to mistakes. The authors created a way to safely update rankings at any point without increasing errors, using mathematical methods that consider all possible score dependencies. This means you can stop testing a model early when confident about its rank or check the leaderboard anytime without losing accuracy guarantees. Their method works even when models are tested on the same data and keeps error rates at a chosen level.
What this means in practice
- •For machine learning evaluation teams: Create leaderboards that can be safely updated item-by-item without inflating error rates or biasing model ranks during ongoing evaluations.
- •For statistical quality control teams: Monitor system rankings in real-time with reliable confidence at any stopping point, even when scores on items are dependent.
Authors
Hamed Khosravi, Xiaoming Huo
Abstract
Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model's rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level $α$, in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models' scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data.