Self-evolving skills often fail to improve on new tasks reliably
Do Self-Evolving Skills Generalize to Held-Out Tasks?
Artificial Intelligence
Summary
This paper looks at AI agents that learn useful steps called skills by practicing tasks repeatedly. The researchers ask if these improved skills actually help with brand-new tasks that the AI hasn’t seen before. They tested several methods and found that many skills don’t carry over their improvements well to new tasks because they sometimes overfit to details specific to old tasks. The authors also propose a new approach, called Generalizable Skill Optimization, which writes a new skill for each task and performs best overall.
What this means in practice
- •For software developers: Improve automated code generation tools by creating task-specific skills rather than relying on reusable evolving skills.
- •For automation engineers: Design more reliable process automation by generating fresh task-specific procedures instead of adapting generalized ones.
Authors
Xihao Piao, Zifeng Wang, Zhen Chen
Abstract
AI agents can externalize what they learn from past tasks into reusable \emph{skills}, such as procedures, checklists, code, or other executable artifacts, that can be retrieved and reused when solving new tasks. Self-evolving skill methods keep rewriting these skills after each round of practice on training tasks, and the skill is then used on new tasks of the same kind. We ask a question: does the improvement a skill shows on its training tasks carry over to new test tasks? We test five self-evolving methods and a one-shot skill on six benchmarks, with the same model, the same agent, and the same train/test split for every method. Of the 21 skills that improve on their training tasks, 5 keep all of that improvement on the test tasks, 13 keep part of it, and 3 keep none of it. No existing method is best everywhere. When we read the skills, the ones that carry over badly often fix details that should depend on the task, such as column names and output files, or turn a fix for one failure into a rule for every task. An LLM judge that reads the skill content can often see this: it ranks finished skills the same way the test results do in 86\% of pairs. But it predicts the effect of a single edit poorly, so edits still have to be tested by running them. Based on these findings, we describe Generalizable Skill Optimization (GSO), which keeps only a guide for writing skills and writes a new skill for each task; it scores highest on all six benchmarks.