Optimizing repository skill documents improves coding agent performance modestly
Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents
Artificial Intelligence
Summary
Coding agents use special text files called SKILLs that explain a repository’s code to help them solve coding tasks. The authors studied how well automatically improved SKILL files help agents handle harder coding challenges taken from real project histories. They found one method, GEPA, can slightly improve agent success, but the improvement is small and may overlap with normal variation. Still, the improved SKILL files contained useful insights that a real project maintainer recognized.
What this means in practice
- •For software engineering teams: Improve coding agent effectiveness by enhancing repository documentation through optimized SKILL files based on historical pull request data.
- •For devops engineers: Use optimized SKILL documents to support automated tools that assist in repository code maintenance and review tasks.
Authors
Mykhailo Kozyrev, Andrei Kozyrev, Anton Podkopaev
Abstract
Coding agents increasingly read repository knowledge from SKILLs --- plain \texttt{.md} files versioned alongside the code. Recent work synthesizes these files automatically, by optimizing the document against a benchmark. A bare repository comes with no benchmark, and the synthetic tasks prior work builds are small enough that a capable agent saturates them with no document at all. We mine harder tasks --- merged pull requests of the repository, reverted at a single frozen base commit; and score a candidate document by whether the same agent does better with it than without it. On three Kotlin repositories, the documents GEPA finds raise this score by $4.9$pp on average, and the ones SkillOpt finds leave it where it started, $0.1$pp above the seed. The GEPA gain matches what prior work reports with the same optimizer, and at the dataset size a single repository supplies it cannot be separated from the agent's run-to-run variance; settling that would take more tasks than one repository's history yields. The documents themselves read better than the score: a maintainer of one repository found in them knowledge one only gets by working in the project.