Agent evaluations depend on tasks and scaffolds for reliability
Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard
Artificial IntelligenceMachine Learning
Summary
Evaluating AI agents to see which is best can be tricky because scores depend not just on the agents but also on how the tests are done. The authors developed a new way to measure how much of the score differences come from actual agent quality versus random noise or test setup. They found that just adding more test tasks doesn't always make rankings more reliable if the way tasks are structured varies a lot. Combining different sets of tests can give better and cheaper comparisons.
What this means in practice
- •For llm deployment teams: Make more reliable model rankings by focusing evaluation resources on diverse test setups rather than just more similar tasks.
- •For ai system evaluators: Use a Bayesian variance analysis method to identify what parts of agent evaluation scores truly reflect model differences.
Authors
Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer, Sanmi Koyejo
Abstract
Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models. We ask which conclusions current agent evaluations reliably support and what additional evaluation would improve them. We develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change rankings. We find: (1) Reliability depends on the measurement goal. Fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying-model reliability is substantially lower (0.148-0.841). (2) Scaffold choice can change conclusions. Inter-scaffold reliability measures whether scaffolds preserve model rankings, showing that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of a benchmark by at most 0.097 when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from 0.44 to 0.75 at the same task budget and can reduce projected cost by up to 83\%. Evaluation design should follow the intended claim: identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.