Leaderboard scores may not reliably show which LLM agent is better
What Does an LLM-Agent Leaderboard Rank Actually Compare?
Artificial Intelligence
Summary
Leaderboards rank AI agents like large language models (LLMs) based on their performance, but these rankings don't always mean one agent is truly better than another. The paper shows that differences in tasks, data sources, and evaluation methods can change how well scores compare agents. The authors propose a careful method to check when it’s valid to say one agent beats another, considering uncertainty and the tests done. They find that small score differences often can't prove superiority, and what counts as better depends on the details of the evaluation.
What this means in practice
- •For ai evaluation teams: Improve how AI system comparisons are made by incorporating uncertainty and task details when interpreting leaderboard scores.
- •For product managers for ai tools: Use clarified evaluation methods to choose which AI agent to deploy based on more reliable performance comparisons.
Authors
Wei-Jung Huang
Abstract
An LLM-agent leaderboard invites a familiar inference: an agent ranked above another is the better agent. Public evaluation logs may not support that conclusion when systems differ in task mixture, label source, release detail, or cost rule. We study what leaderboard scores estimate and when they justify pairwise superiority conclusions. Our estimand-aware pairwise procedure states the comparison target and measurement source, checks common support, and evaluates the supported difference using a stated uncertainty rule and practical margin. Controlled checks evaluate the decision labels under known finite-sample conditions and show why uncertainty must be included when judging sensitivity to target reweighting. Across SWE-bench, AgentRewardBench, and tau2-bench, close rank differences are often unresolved; proxy labels and utility rules can also change which system is selected. DataAgentBench and Open Agent show what remains estimable from coarser public records. A leaderboard score summarizes a released evaluation, whereas a fine-grained superiority claim additionally depends on the estimand and uncertainty rule used to interpret the difference.