Coding agent leaderboards struggle to rank top systems accurately
Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead
Software EngineeringArtificial Intelligence
Summary
Leaderboards rank computer programs that write code, but very close scores don’t always mean clear performance differences. The authors found that many top-ranking systems share most successes and failures, making it hard to say which is truly better. They suggest new ways to compare these systems by focusing on shared outcomes and reliability of differences instead of small score gaps.
What this means in practice
- •For software development teams: Use the audit protocol to better understand which coding tools truly have different strengths before choosing among similar top performers.
- •For machine learning operations teams: Report measurement details and resolution capacity of code-writing models to avoid misleading rankings in deployment decisions.
Authors
Fengshuo Liu, Ying Liu, Ruize Sun, Lie Luo, Siyuan Guo
Abstract
Small differences on coding-agent leaderboards are often read as an ordering of systems. We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models. On Verified, the leading two entries each resolve 396 of 500 instances. The top ten share 285 successes and 51 failures, leaving 164 instances that distinguish their outcomes. Frontier solution sets have median nesting 0.935 against a score-implied baseline of 0.774, indicating strongly shared successes. Scores also depend on the evaluated model-scaffold pair: observed within-model scaffold ranges reach 29.8 percentage points, compared with the 8.8-point spread of the top thirty. Six of nine cell-mean interaction tests remain significant after Holm correction, although this observational design does not identify causal scaffold effects. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05, while the larger Test split separates 14 of 23. A stated leader-based rule yields three descriptive tiers, or two after Holm correction; non-rejection does not establish equivalence. We release the partition and a five-step audit protocol that profiles shared outcomes, tests paired differences, reports grouping sensitivity, and estimates the instance budget needed for resolution. The results motivate reporting comparison-set-specific resolution and model-scaffold provenance instead of interpreting small aggregate gaps as established rank differences.