Re-executing llm coding tests confirms patch grading stability and unresolved risks
Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk
Software EngineeringComputation and Language
Summary
The paper looks at how tests that grade code written by large language models (LLMs) perform when run again. The authors found that most decisions about which code patches are accepted or rejected stayed the same upon re-execution, showing grading is generally stable. However, there remains some uncertainty about whether some patches might be mistakenly accepted. Their work highlights the importance of careful testing but also shows that some risks in automated grading decisions remain unresolved.
What this means in practice
- •For machine learning engineers: Verify and audit grading setups for LLM-generated code to ensure stable acceptance decisions and detect potential grading risks.
- •For software testing teams: Use re-execution of automated coding tests to confirm the reliability of code acceptance in continuous integration pipelines involving LLM coding agents.
Authors
Tianyu Ding, Yaocheng Tong, Muzhi Yang, Fei Jiang, Lei Yang, Boya Yu
Abstract
Executable tests grade large language model (LLM) coding-agent patches for leaderboards, rewards and self-improvement. Two preregistered re-execution studies returned negative results. SWE-bench Pro v1's offline option rejected none of 59 accepted Go patches whose grading configuration changed. Their exploratory, unweighted upper confidence limit is 4.95% under candidate independence, leaving a 3% exposed-patch risk unresolved; the preregistered estimator uses inverse sampling-probability weights. An extension was inconclusive. In a selected stratum, a reconstructed module cache built from the reference solution's dependencies rejected accepted alternatives from four models that passed with their own dependencies. Re-executing 1,929 selected agent-version/task evaluations from the Darwin Gödel Machine's archive preserved best-agent selection and system-variant ranking, with search history and scores of evaluations not rerun fixed. Exploratory host, image and adapted-grader comparisons also preserved them. Seven of 23 disagreements with archived verdicts did not recur on a second host; rebuilt images changed 16 three-run majority verdicts (0.83%) under four-CPU execution. We provide an evidence-to-claim summary, lessons for grader audits and an anonymous replication package. These results concern accepted patches and fixed-history decisions; general policy safety and adaptive-learning robustness require broader evaluations.