Scanner detects safety issues in AI agent skills before installation
Scan Before You Install: Detecting and Mitigating Local and Composite Vulnerabilities in Skills
Cryptography and SecurityArtificial Intelligence
Summary
AI agents use special tasks called skills, which can run code or access files. Sometimes these skills have hidden safety problems that can cause harm. The authors created a scanner that checks skills for both simple and complex safety issues, even when skills interact with each other. They tested the scanner on many skills from public sources and found it to be accurate at finding problems and suggesting fixes that keep things working safely.
What this means in practice
- •For software security teams: Scan AI agent skills before deployment to detect hidden vulnerabilities and prevent unsafe execution.
- •For devops engineers: Identify and fix risky skill interactions in multi-skill systems to maintain safe automation workflows.
Authors
Parth A. Gandhi, Avital Aviv, Prasanna N. Wudali, Yuval Elovici, Asaf Shabtai
Abstract
An agent skill packages natural-language instructions, scripts, and reference files that an LLM-based agent loads for a task. Skills may access the file system, shell, and network under user-authorized permissions; weaknesses can lead to unsafe execution. Existing scanners primarily target malicious skills, not unintended vulnerabilities in public repositories, and analyze skills in isolation, overlooking multi-skill interactions. To address these gaps, we developed a six-module scanner: Content Extraction, Local Analysis, Composite Analysis, Validation, Reporting, and Mitigation. Local Analysis detects unintended vulnerabilities in benign skills. On a 556-case leaderboard subset of SkillTrustBench, it achieved 95.90% recall with a 4.22% false positive rate; on a balanced sample of 1,000 MalSkillBench skills, its F1 was 0.947, outperforming Cisco's Skill Scanner, NVIDIA's SkillSpector, AgentSkillsScanner, and SkillSieve. In 1,587 skills from the 75 most-starred GitHub repositories, three analysts confirmed 53 of 59 findings (89.8% precision) across 23 repositories. We developed proof-of-concept (PoC) attacks for each vulnerability, disclosed 10 medium- and high-severity cases to maintainers, and tested mitigations on the remaining cases. Composite Analysis models dependencies among skills through shared resources, invocations, and instruction-mediated interactions. On SCR-Bench, its F1 score was 0.991 (98.7% precision, 99.6% recall), versus 0.816-0.858 for three SkillsGuard configurations released with CompoSkill. It identified two composite findings on GitHub, validated by PoC attacks. The Mitigation module recommends skill substitution, patches, and configuration changes. For 47 of 55 GitHub findings, mitigations prevented PoC attacks in all trials while preserving tested legitimate functionality.