Coding agents review and change their setup with mixed success
How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation
Software EngineeringArtificial Intelligence
Summary
Coding agents can look back at their past work sessions and suggest changes to the software and instructions they use to do their jobs. The authors studied what kinds of changes these agents suggest and tested whether those changes help reduce repetitive work later on similar tasks. They found that some changes helped, some made no difference, and others even caused extra work. The study shows that for these self-changes to really improve performance, agents need ways to check if changes actually help and to undo them if they don’t.
What this means in practice
- •For software development teams: Improve development workflows by integrating agent-driven reviews that suggest harness changes to reduce repetitive coding problems across tasks.
- •For automated testing teams: Use retrospective harness adaptations to fine-tune test configurations and lower repeated test failures on similar code bases.
Authors
Christoph Treude, Sebastian Baltes
Abstract
Coding agents can adapt their harnesses, the software, configuration, and instructions used to run them, based on experience from earlier tasks, but whether these adaptations improve their behavior on later tasks remains unclear. We study \retro, a skill that asks an agent to review a completed session and propose harness changes, and ask what changes it proposes and whether implemented changes reduce repeated work on later tasks. We ran Claude Code on 30 SWE-bench Verified issues, qualitatively coded the 125 proposals from \retro, implemented 24, and evaluated 12 harness changes on later issues from the same repositories in fresh sessions with and without each change. Most proposals referred to an event in the session, but 18 of 43 lint-related proposals recommended a practice regardless of what had happened. Among the seven evaluations in which sessions without the change encountered the targeted problem, two changes eliminated the targeted action, one reduced its occurrence, four showed no reduction, and some changes added work on unrelated issues. Harness self-evolution requires more than generating and implementing changes: it needs ways to establish whether a change helps on later tasks and to revise or remove changes that do not.