SWE-Bench Pro V2 offers tougher and cleaner coding tasks for AI agents

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

Software EngineeringArtificial Intelligence

Summary

Coding AI agents need tests to see how well they solve complicated programming tasks. The previous version of SWE-Bench Pro had problems because some tasks became too easy or could be tricked by clever cheating. The authors fixed or removed many tasks, checked every instruction carefully, and created rules to stop agents from gaming the system. Their updated version keeps tests fair and challenging as AI agents get better.

What this means in practice

  • For software engineering teams: Use SWE-Bench Pro V2 to evaluate AI coding agents on hard, verified software development tasks that reflect real-world programming challenges.
  • For ai platform developers: Integrate SWE-Bench Pro V2’s secure and precise benchmarking method to accurately measure agent coding improvements without exploitation.

Authors

Soham Dan, Jeff Da, Sanyam Bhutani, Miguel Romero Calvo, Ivan Guevara, Fernando Carabedo, Jose Bernal, Julio Reyna, Luis Leandro Monsalve Mendez, Ying Liu, Daniel Yue Zhang

Abstract

SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.