Skill and policy changes cause self-learning agents to fail over time

Evolving Together, Failing Apart: The Skill-Policy Co-evolution Trap in Self-Improving Agents

Artificial IntelligenceComputation and Language

Summary

When AI agents learn by improving their skills and decision rules together, they can perform well at first. But if their skills change after training without updating their decision rules, their performance often drops a lot. The authors found that this problem is built into how these agents work, and it gets worse the longer they train together. They created a new way to test agents by simulating how skill changes affect them, aiming to better measure real-world performance after deployment.

What this means in practice

  • For ai system developers: Evaluate and improve AI agents’ robustness to changes in their skill set after deployment using the CODA protocol introduced in this paper.
  • For software reliability engineers: Use CODA metrics to audit AI components’ ability to handle evolving or corrupted modules in complex software systems.

Authors

Yujie Feng, Xiaobin Wang, Jinluan Yang, Xiaonan Shi, Pengjun Xie, Philip S. Yu, Xiaoming Wu, Jieping Ye

Abstract

Skill-policy co-evolution, in which an agent's policy model and its skill bank improve each other during training, is a promising paradigm for self-improving agents. Current evaluations score the co-evolved policy on the final bank it trained with (the delivered bank). After deployment, however, skills may be rewritten, added or removed more frequently than the policy model is retrained, leaving the policy to operate on a drifted skill bank that neither the training objective nor existing evaluation protocols address. In this paper, we first establish, through theoretical and empirical analysis, that the fragility hidden by this blind spot is not incidental. The acceptance rule these methods share (cooperative acceptance) ties a policy's competence to the bank it co-evolved with, leaving it fragile once skills drift, regardless of the implementation; empirically, sensitivity to skill drift grows as co-evolution training proceeds. We therefore introduce the CO-evolution Drift Audit (CODA), a deployment-oriented evaluation protocol that simulates skill drift and, alongside success rate under worst-case drift, introduces two new metrics: a consistency rate under meaning-preserving edits and a resistance rate to corrupted skills. Across four co-evolution methods and five benchmarks (six settings), worst-case success is 5-13 points below the peak measured for each delivered pair, erasing 52-93% of the delivered bank's gain over the no-skill baseline. CODA thus reframes the target of co-evolution, from peak performance on the co-evolved bank to the ability to harness skills however they evolve, and provides a shared standard against which future methods can be audited.