SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills
2026-07-10 • Cryptography and Security
Cryptography and SecuritySoftware Engineering
AI summaryⓘ
The authors present SkillLogic, a method to analyze how instructions within agent skills relate logically and to test if these relations are followed correctly. They identified eight types of relations like conditions that must be met before actions or fallback steps after failures. By studying over 5000 skills, they found logical relations are common, and created a benchmark called SLBench to evaluate if AI language models follow these rules safely. Their tests showed many unsafe behaviors, but a tool called SLGuard helped reduce mistakes. Overall, the authors highlight that ensuring agents follow logical rules in skills is an important safety challenge.
LLM agentsagent skillslogical relationspreconditionsconstraintsfallbacksbenchmarkCodexClaude Codesafety
Authors
Xuan Chen, Chengpeng Wang, Lu Yan, Xiangyu Zhang
Abstract
Agent skills extend LLM agents with reusable procedures, tools, and domain-specific workflows, but their safety depends on resolving dependencies among interacting instructions. We introduce SkillLogic, a framework for analyzing logical relations in skill files and constructing executable tests from them. Our taxonomy covers eight relation types, including preconditions that gate valid actions, constraints that limit how allowed actions may be performed, and fallbacks that specify recovery behavior after failure. Using SkillLogic, we scan over 5000 public skills and find that 70% contain at least one logical relation. We then construct SLBench, an 86-case executable benchmark from high-confidence, high-impact, and locally testable relations. Evaluating Codex and Claude Code across six LLM backbones shows unsafe rates up to 70%, with violations leading to privacy leaks, unsafe configuration changes, and incomplete cleanup. The human audit attributes failures to both agent capability gaps and low-salience skill text. We further show that SLGuard, a lightweight inference-time scaffold, reduces violations by 63% on targeted cases. Our results establish logical-relation following as a distinct reliability challenge for skill-guided agents.