Coding agents struggle to keep skills after learning new tasks
Can Coding Agents Sustain Continual Evolution without Forgetting?
Software EngineeringArtificial Intelligence
Summary
The paper studies whether coding agents can keep improving themselves on many tasks without forgetting old skills. The authors found that when agents learn to do a new programming task, their performance on earlier tasks often drops a lot. This problem is called catastrophic forgetting, and it limits how coding agents can evolve continuously over time. The authors tested different scenarios and methods and shared insights to help future improvements.
What this means in practice
- •For software development teams: Improve automated coding assistants by understanding how they forget previous task knowledge when adapting to new programming challenges.
- •For ai system integrators: Design better multi-task AI tools by acknowledging and addressing forgetting when evolving coding agent capabilities over time.
Authors
Shangziqi Zhao, Yitong Zhang, Jia Li
Abstract
Recently, harness self-evolution has emerged as an important paradigm for improving agent performance. Prior studies show that evolving the agent harness can effectively enhance agent capabilities without modifying the foundation model's parameters. We find that existing approaches largely study agent evolution on a single task. In practice, coding agents may need to handle multiple task types over time. Whether an agent harness can continuously evolve across multiple tasks without suffering from catastrophic forgetting remains an important open question. In this paper, we investigate this open question through a comprehensive empirical study of continual harness self-evolution. We organize three software engineering task datasets into six different evolution directions and apply two representative harness evolution methods to perform continual evolution along these directions. Our experimental results show that evolution on a subsequent dataset can substantially undermine the performance on previously seen datasets. For example, with DeepSeek-V4-Flash-0731, Meta-Harness improves its performance on SWE-Bench to 94.0% after evolution. However, after further evolution on FeatureBench, its performance on SWE-Bench drops to 32.0% by the fifth round, corresponding to a 66% relative decrease. Finally, based on our experimental findings, we identify several insights, providing directions for the continual self-evolution of agent harnesses. Code is available at https://github.com/THU-Agent/continual-harness-evolution.