Benchmarks become less reliable with more feedback criteria
How Reusable Are Benchmarks with Richer Feedback?
Machine Learning
Summary
Benchmarks are tools used to compare how well computer models perform tasks. This paper shows that when developers use many criteria to get feedback on models, it becomes much harder to pick the best one accurately. The authors found that as the number of criteria grows, the amount of test data needed to confidently choose the best model grows very quickly. They also found that limiting feedback to certain types of performance comparisons can cause wrong conclusions about which model is best. This challenges previous assumptions about how developers respond to benchmark feedback in complex situations.
What this means in practice
- •For machine learning developers: Identify situations where relying on many feedback criteria may lead to misleading model comparisons and require larger test sets to avoid false best model selections.
- •For ai benchmarking platform teams: Design benchmark feedback systems that avoid feedback modes prone to generating false winners by carefully controlling criteria and evaluation profiles.
A theory result. No direct application yet.
Authors
Youssef Allouah, John Duchi
Abstract
We study whether benchmarks reliably guide model selection as developers adapt to evaluation feedback across multiple criteria. We find that the worst-case test-set size needed to estimate the best score among $k$ adaptively chosen models, under any convex combination of the criteria, grows exponentially with the number of criteria, reaching the $Θ(\sqrt{k})$ cost of answering $k$ adaptive statistical queries with only $O(\log k)$ criteria, at fixed accuracy and confidence. In attacks on multi-task large language model benchmarks with five to ten criteria, feedback restricted to nondominated task profiles produces large reused-to-held-out score gaps and frequent false winners. These results challenge a prominent explanation for prior observed reliable benchmark reuse---that developers mainly respond to convincing improvements over the current best---in rich-feedback settings, while leaving open how often ordinary model development encounters this vulnerability.