LLM program repair results need clear experiment details
Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation
Software Engineering
Summary
Fixing computer programs automatically with AI models can be done in many ways, but how the repair task is set up affects the results a lot. The authors looked closely at how different studies provide input data, allow tool use, show feedback, and check fixes, finding these factors are often unclear. They created a system that clearly describes these settings so others can understand and compare results better. This helps make research on AI program fixing more repeatable and fair.
What this means in practice
- •For software developers: Use detailed experiment descriptions to replicate and trust AI repairs in software projects more effectively.
- •For devops teams: Compare AI-based repair tools accurately by understanding their different experimental inputs and capabilities.
Authors
Xushu Dai, Yicheng Cai, Nanqing Luo, Pei-Yu Tseng
Abstract
Evaluations of automated program repair (APR) systems commonly report the benchmark, the number of repaired defects, and the tests used for final patch validation, but these items no longer fully specify the repair task presented to a system. Recent LLM-based systems differ in the information supplied before repair, the repository and testing operations permitted during repair, and the feedback returned after unsuccessful attempts, allowing the same benchmark to instantiate substantially different repair tasks ranging from localized patch generation to repository-level diagnosis and iterative repair. We present a framework for explicitly specifying the experimental settings associated with reported APR results. We analyze reported experimental settings from systems evaluated on Defects4J and SWE-bench and characterize each result by its task unit, fault-localization assumptions, initial input, tool access, repair-time feedback, final validation, and resource budget. Our analysis shows that benchmark identity alone is insufficient to reconstruct the evaluated task or determine the appropriate scope of comparison across reported repair rates. We therefore introduce a machine-readable schema for specifying each experimental setting to improve reproducibility and make the scope of cross-system comparisons explicit.