Agent skill testing reveals limits of plugin migration advice
Evaluating Agent Skills for Version-Specific Plugin Migration: A Retrospective Study
Software Engineering
Summary
Keeping software plugins up to date is tricky because new versions have different rules. The authors studied how an AI helper performs advice on moving plugins to new versions by examining many test cases and detailed reports. They found the AI improved scores somewhat but had errors that gave too much credit in some cases, and other reviewers saw similar patterns. The work links overall performance scores to detailed checks, helping future improvements but leaving actual automatic fixes and new testing methods for later.
What this means in practice
- •For software maintenance teams: Assess migration advice tools by tracing decisions to compatibility contracts and checking grading consistency.
- •For software testing teams: Use detailed report reviews and executable checks to improve the reliability of plugin upgrade evaluations.
Authors
Beiming Liu, Haihao Li, Minjie Chen, Ning Chen, Yiran Wang, Jiming Ye, Puzhao Zhang, Tongtao Wang, Sheng Gao, William Jin, Weihao Mu, Chengzhi Liu, Yucheng Xia, Guangren Wang, Chaoyang Fan, Changfeng Huang, Xunming Lin, Yuanjie Shen
Abstract
Agent skills package version-specific maintenance knowledge for coding agents, but a higher diagnostic score does not by itself show that the resulting migration advice satisfies the target version's contract. We study a shipped plugin-upgrade skill through an archive of 64 reports on 16 static migration tasks, with two attempts per condition and 328 criterion decisions. With the skill, mean recorded reward rises from 93.83 to 98.75, a gain of 4.92 points (95% task-bootstrap interval [0.31, 10.86]); the gain is concentrated in one task, and eight task pairs are at the ceiling. Tracing every decision to its contract domain and reviewing ten reports in depth exposes grading errors that favor either arm; in one, a containment predicate that accepts the parent directory still receives full credit. Executable probes confirm this defect and show that a working teardown repair is excluded only by a narrower lifecycle rubric. Replacing the reviewed decisions keeps the estimate positive (4.61 to 5.39 points) but moves its interval to or across zero. Re-grading all 64 reports with judges from two other model families, without arm labels or prior scores, agrees with the original judge on 91.8% and 95.7% of decisions (weighted $κ=0.64$ and $0.72$) and gives gains of 10.63 and 6.09 points. The study contributes a traceable evaluation that connects aggregate reward to contract-level evidence and judge sensitivity, together with concrete review checks for migration advice. Executable end-to-end repairs, independent human annotation, and other frameworks are left to future work.