Compact language models detect malicious skills in AI assistants
SKILLLITE: Evidence-Guided Malicious Skill Auditing with Compact LLMs
Cryptography and SecurityArtificial Intelligence
Summary
AI assistants are getting smarter by using special add-ons called Skills, but some of these Skills can be harmful by doing sneaky bad things. The authors found that smaller, local AI models have trouble spotting these bad Skills because the harmful actions are hidden and complex. They created a new method called SKILLLITE that helps small AI models find harmful behaviors by first pulling out important clues and then deciding if the Skill is bad. This method works better than other approaches and is fast enough to be used in real situations.
What this means in practice
- •For ai software developers: Detect harmful behaviors in third-party AI assistant Skills before deployment using compact models for better security and privacy.
- •For cybersecurity teams: Audit AI agent extensions locally to identify and block malicious functionalities quickly in resource-limited, sensitive environments.
Authors
Haoran Ou, Gelei Deng, Xuanye Zhang, Wenbo Guo, Tianwei Zhang, Kwok-Yan Lam
Abstract
As LLM-based agents perform increasingly complex tasks, Agent Skills have emerged as a flexible mechanism for extending their capabilities. An Agent Skill packages task-specific instructions with executable components and auxiliary resources to provide specialized functionalities. However, the growing adoption of third-party Skills introduces a new supply-chain attack surface. Malicious Skills can embed harmful behaviors that abuse agent privileges and compromise the agent execution environment or accessible resources. Although recent LLM-based malicious Skill auditing approaches have achieved promising performance, they often rely on capable commercial LLMs. How to achieve effective auditing with compact, locally deployable LLMs in security-sensitive and resource-constrained settings remains largely unexplored. Our investigation reveals that compact LLMs struggle to identify malicious behaviors hidden in complex Skill packages. This difficulty arises from both the implicit nature of such behaviors and the limited reasoning capacity of compact LLMs. To address these challenges, we propose SKILLLITE, an evidence-guided agentic framework for malicious Skill detection. SKILLLITE effectively extracts security-relevant behaviors and infers the intended functionality from complex Skill packages. It then employs a compact LLM to assess the maliciousness of the Skill based on the observed behaviors and their functional context. Experiments show that SKILLLITE improves malicious Skill detection across different compact LLM backbones and outperforms existing representative auditing baselines. Its effectiveness generalizes to behaviorally confirmed in-the-wild malicious Skills. Meanwhile, SKILLLITE maintains a low inference latency, supporting its practical deployment.