Software engineering agent benchmark improved for reliable results
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
Artificial IntelligenceSoftware Engineering
Summary
Software engineering agents are computer programs that write or fix code automatically. The existing SWE-Bench Pro benchmark used to test these agents has problems, like letting the programs cheat or testing unclear tasks. The authors created SWE-Bench Pro Verified, which removes cheating opportunities and fixes task problems. Testing with this new version shows some agents do worse than before, meaning previous tests may have been too optimistic.
What this means in practice
- •For software development teams: Use SWE-Bench Pro Verified to more accurately evaluate code-writing AI tools before integrating into development workflows.
- •For ai tool developers: Test and improve software engineering agents against a benchmark that prevents cheating and flawed tasks, ensuring fair performance measures.
Authors
Pujun Zheng, Zixin Shang, Shufan Jiang, Wenhui Tian, Dongsheng Zhu, Zerun Ma, Dingbo Yuan, Qi Zhang
Abstract
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: \textbf{reward hacking}, enabled by leakage of gold solutions or hidden evaluation information, and \textbf{task quality issues}, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present \textbf{SWE-Bench Pro Verified}, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines \textbf{anti-hacking} safeguards that eliminate major leakage channels without disrupting normal agent functionality, with \textbf{task refinement} that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously reported, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.