Software agent scores recovered accurately using fewer benchmark tasks
HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference
Artificial Intelligence
Summary
Evaluating AI software agents on all benchmark tasks can be very time-consuming and costly. The authors introduce a method called Hierarchical Score Posterior Recovery (HSPR) that predicts an agent's overall score using only a small number of tasks. It uses previous results from similar agents and smartly picks which tasks to test next to better estimate scores. Their method reduces errors significantly even when testing only 10%-20% of the tasks.
What this means in practice
- •For machine learning engineers: Reduce the cost of evaluating new AI agents by accurately estimating benchmark scores from fewer test cases.
- •For quality assurance teams: Prioritize testing efforts on software components where uncertain performance estimates can be improved efficiently using adaptive task selection.
Authors
Wanyi Chen, Wanqin Shen, Yuanchen Shi, Zhensu Sun, Fang Kong
Abstract
Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful structure, but tasks with identical source responses can have different outcomes for a new target agent. We introduce Hierarchical Score Posterior Recovery (HSPR), which combines source-fitted response factors with uncertain group success rates. Observed target outcomes update the posteriors over the target latent state and group rates; posterior-guided acquisition selects subsequent tasks. Its score-based criterion equals the expected one-step reduction in full-score posterior variance. Observed successes and posterior-predictive counts for unqueried tasks yield a full-score posterior. The final estimate uses its median with source-selected calibration. At a 10% task budget on four task banks from the SWE-bench family, HSPR achieves the lowest mean absolute error (MAE) on three banks and the lowest maximum absolute error (MaxErr) on all four among the methods compared. Removing group uncertainty or replacing adaptive queries increases macro MAE in the respective controls. Macro MAE decreases across 5-20% budgets.