Signal or Noise? A Benchmark Study of Agent Skills in Web Development
2026-08-24 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how adding reusable code helper modules called Agent Skills affects coding AI agents working on web development tasks. They created a special test (WebDev-Skills-Bench) to see if these Skills actually help or just make the task prompts longer and harder. They found that adding Skills often made the agents slower and less successful overall, with benefits in only a minority of cases. The impact varied by the specific model and task, showing that Skills should be carefully chosen for each situation rather than added automatically. The authors suggest evaluating these Skills with controls that account for prompt length and model differences.
Agent Skillsprompt lengthweb developmentbenchmarkingPass@2task completioncoding agentsprompt engineeringmodel evaluation
Authors
Ziyue Yang, Fan Ding
Abstract
Agent Skills are reusable procedural modules that are increasingly injected into coding-agent sessions to encode framework conventions, anti-patterns, and reusable tools. However, because each injected Skill expands the prompt of every query, an effective Skill benchmark must determine not only whether an agent can solve a task, but whether the Skill should have been injected at all. We introduce WebDev-Skills-Bench and use it for a controlled empirical study of 31 public WebDev Skills on 50 Web-Bench projects and 1,000 ordered tasks. The benchmark compares four matched conditions, including a length-matched irrelevant control and leave-one-out component ablations. To isolate Skill effects from prompt-length artifacts, we place only SKILL.md in the prompt while mounting auxiliary files into the agent workspace. Across four models, target Skill injection reduces mean Pass@2 by 1.3% to 4.2%, lowers task completion depth, and increases token cost by 72% to 394%, with gains in only 17% to 36% of Skill-project pairs. Length-matched controls reveal two failure modes: some models are length-distracted, where an equally long irrelevant Skill reproduces most of the loss, while others are content-misled, where prompt length is neutral but Skill content still lowers Pass@2 by 1.1% to 1.4%. Further analysis shows that losses concentrate on easy early tasks, Skill rankings transfer weakly across models, and anti-pattern rules outperform example-heavy content within helpful Skills. These findings recast a matched Skill as a hypothesis about a particular Skill-project-model triple rather than a portable asset, reframing injection as a per-deployment routing decision and making length-matched controls and per-model audits a minimum standard for Agent-Skill evaluation.