SkillDelta predicts when skills improve task performance effectively
When Does a Skill Add Value? Task-Conditional Gain Prediction for Selective Skill Use
Artificial Intelligence
Summary
Sometimes adding special skills to computer programs helps them do tasks better, but other times it doesn't or even makes things worse and slower. The authors created SkillDelta, a way to guess ahead of time if a skill will actually help on a new task by comparing past results from the same program with and without the skill. This helps use skills only when they add value, saving effort and improving success rates on many test tasks. The method works best by deciding which groups of tasks benefit from skills.
What this means in practice
- •For software developers: Optimize resource use by predicting when activating extra skills improves software task success.
- •For automation engineers: Enhance automation systems by selectively enabling skills only when they positively impact task outcomes.
Authors
Anjie Xu, Zhiyu Zhang, Ruiqing Ding, Fengli Xu, Leye Wang
Abstract
Agent skills are expected to improve task performance. Yet we find that they often provide no benefit, and can even hurt performance while incurring additional token costs. Can we predict whether a skill will help before the agent acts? We introduce SkillDelta, a framework for predicting task-conditional skill gains from paired executions of the same agent with and without the skill. A local predictor transfers these historical gains to new tasks without retraining the agent. Under explicit transfer assumptions, our analysis links support coverage, representation mismatch, and execution noise to prediction error and decision regret. Across five benchmarks and three target agents, paired history improves observed-gain ranking over skill-assisted outcomes alone in 12 of 15 settings. At matched expected skill-use rates, SkillDelta improves success over random activation in all 15 settings, with an average absolute gain of 4.3%. Most of this advantage comes from allocation across task groups. Evidence for additional within-group selection value is strongest on ToolQA and weaker elsewhere. Code is available at https://github.com/TankTechnology/skilldelta.