Training coding agents by evolving their tools and strategies together
Training the Whole Agent, Not Just the Model
Software Engineering
Summary
Training AI systems that write code usually focuses on improving just the AI’s decision-making, keeping the environment it uses fixed. The authors found that by also adapting the environment—the tools and ways the AI interacts with tasks during training—they could help the AI learn better and fix mistakes more effectively. They developed a method that changes these tools as the AI improves, leading to higher success rates in coding tasks. This approach creates a matched pair of the AI and its tools that work best together.
What this means in practice
- •For software engineering teams: Improve automated code generation tools by jointly training AI and their execution environments to boost code correctness rates.
- •For cloud platform operators: Enhance reinforcement learning agents deployed on cloud coding workloads by evolving their operational interfaces along with their policies for better reliability.
Authors
Lingxiao Tang, Wei Tao, Yiwen Guo, Lingfeng Bao
Abstract
Training coding agents typically updates the policy while keeping the harness fixed. During training, trajectories reveal flawed repair strategies and interaction problems that vary across tasks and change as the policy learns. We introduce HarnessAdapt-RL, which trains the policy and adapts its harness during reinforcement learning. For flawed repair strategies, a frozen teacher generates task-specific mechanism-level skills. For interaction problems recurring across tasks, a separate teacher model evolves tools and hooks, keeping each change only if it improves the observed pass rate or the observed trainable rate without lowering either. The policy trains with the adapted harness, and its new trajectories guide further adaptation. With a 9B base model, HarnessAdapt-RL improves pass@1 over GRPO by 9.8, 12.6, and 6.4 percentage points on SWE-bench Verified, Multilingual, and Pro, and outperforms GRPO with the human-designed OpenHands harness. Gains from mechanism-level skills persist after their removal at evaluation. Re-evolving tools and hooks as the policy changes outperforms freezing early versions. Removing them at deployment, in contrast, costs up to 7.7 percentage points. Training therefore produces a model-harness pair: the trained policy and the evolved tools and hooks, with skills removed at deployment.