Agent skills improve smart contract audits with model impact
Demystifying Agent Skills for Smart Contract Auditing: Design, Effectiveness, Behavioral Impact
Software Engineering
Summary
Smart contracts are digital agreements that need to be checked for security risks. The authors studied how special tools called 'skills' help AI agents find vulnerabilities in these contracts. They gathered many real audit skills and tested them with different AI models. They found that the AI model itself matters more than the skill design, and that activating these skills correctly is challenging. The results provide a better understanding of how AI skills work in smart contract security.
What this means in practice
- •For smart contract developers: Use AI agents with advanced models and skill sets to better detect security flaws before deployment.
- •For blockchain security firms: Improve automated audit tools by incorporating diverse audit skills and focusing on effective skill triggering to enhance vulnerability detection.
Authors
Cuifeng Gao, Juantao Zhong, Jiachi Chen, Shuai Wang, Daoyuan Wu
Abstract
LLM agents, notably Claude Code and OpenAI Codex, are emerging as versatile tools beyond coding agents only. These agents can be enhanced with skills---reusable artifacts that package domain knowledge, workflows, and tool-use instructions. To date, however, little is known about how such skills are designed or how they affect agent effectiveness and behavior in practice. In this paper, we investigate these questions in smart contract security auditing, a domain in which agents have shown substantial promise. We systematically collect 83 smart contract audit skills from the wild and evaluate them on EVMBench across seven agent--model configurations. Our study examines three dimensions: (i) the design characteristics of audit skills, including their structure, knowledge representations, workflows, and tool dependencies; (ii) their effectiveness in improving vulnerability detection; and (iii) their influence on agent execution trajectories. We find that audit skills are mostly lightweight but heterogeneous in design, covering a broad yet imbalanced range of vulnerability types. Their effectiveness is determined primarily by the model rather than the agent harness: Codex/GPT-5.5 achieves the largest gains, improving detection score by 22.8% and captured award by 43.2%. We further find that skill triggering is a key bottleneck. When triggered, skills preserve a shared six-stage audit workflow while exhibiting distinct loading patterns and differential effects on agent behavior across configurations. We release our skill corpus and artifacts to support future research.