Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
2026-09-01 • Software Engineering
Software EngineeringArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors address the challenge of evaluating software engineering AI agents, which is usually expensive because it involves multiple steps like writing and testing code. They propose PTA-IRT, a new method that not only looks at whether tasks pass or fail but also considers the detailed steps agents take to solve problems. By using this extra information, their approach better estimates agent abilities and selects tasks more effectively, especially when only a small number of tasks can be used for evaluation. Their experiments show that PTA-IRT outperforms previous methods on multiple software engineering benchmarks.
software engineering agentsbenchmark evaluationitem response theoryexecution trajectoriesprivileged informationcalibration subset selectionability estimationcode explorationtest executionranking recovery
Authors
Kefeng Duan, Dewu Zheng, Yanlin Wang, Xiwen Wang, Ensheng Shi, Xilin Liu, Yuchi Ma, Jiachi Chen, Mingwei Liu, Zibin Zheng
Abstract
Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT.