Coding agents fix performance issues but do not always deliver improvements
Merged, Not Measured: An Empirical Study of Performance Issues Fixed by Coding Agents
Software Engineering
Summary
Sometimes computer programs called coding agents make changes that claim to speed up software. The authors studied over a thousand such performance fixes across many projects. They found that only about half of these fixes were accepted and merged by the software maintainers. Even among accepted fixes, many did not actually improve performance when tested. The decision to accept a fix depended more on the history of the agent in that project than on whether the fix truly worked.
What this means in practice
- •For software maintainers: Help maintainers recognize that merging agent-generated fixes does not guarantee actual performance improvements and encourages cautious code review practices.
- •For software engineering teams: Guide teams in tracking agent history and merge behavior to better predict when automated fixes are reliable.
Authors
Zhenyu Qi, Haotang Li, Jinfu Chen, Huashan Chen, Yutong Zhao, Derui Zhu, Tomas Cerny, Bo Liu, Sen He
Abstract
Coding agents open pull requests (PRs) that claim to speed up software, but studies of human performance fixes say little about how maintainers respond to such a fix or whether its claim holds. From the 71,677 agent PRs of AIDev v4, a text filter and codebook coding by language models and by the authors select 1,262 performance issues fixed by six agents in 582 repositories. We code each issue and its tests and re-execute 23 rejected and 30 merged fixes. (1) 57% of closed fixes are merged, 61% of rejections give no stated reason, and only 6 of the 23 re-executed rejected claims held under our three-run pilot on mostly agent-built workloads. (2) Acceptance rises with the agent's track record in the repository (31-37% to 70%) and with the repository's pre-opening merge rate on its other agent PRs (33% to 84%). Merged fixes delete a larger share of the lines they change (0.26 versus 0.15), a difference that holds within agent and within repository, with no such difference detected in the coded content, description, tests or measurements. (3) Repeated computation and redundant data processing cause 44% of the issues, and 46% of fixes are architectural-level. (4) Agents change tests in 37% of fixes and 11% carry a performance test or benchmark; of the 30 merged fixes, 18 met our delivery criterion, 3 fell short of the claim, 9 showed no significant gain or regressed, and 14 change behavior on untested inputs. The outcome tracks the repository's history with the agent rather than the coded content of the fix, and a merge does not show that the fix delivers what it claims.