SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents
2026-06-01 • Cryptography and Security
Cryptography and SecurityArtificial Intelligence
AI summaryⓘ
The authors created SeClaw, a new system to test the security of autonomous AI agents that use tools and memory. Instead of just checking if the final answer is safe, SeClaw watches the whole process the agent follows to spot risky behavior. It builds security tests automatically from clear rules and runs these tests in a standard environment to fairly compare different agents. This helps researchers better find and fix security problems in these smart AI systems.
autonomous agentslarge language modelssecurity evaluationtask synthesisstateful environmentsexecution tracerisk assessmentbenchmarkingdockersafety-risk scenarios
Authors
Hao Cheng, Changtao Miao, Tianle Song, Yin Wu, He Liu, Erjia Xiao, Junchi Chen, Xiaoyu Shi, Yichi Wang, Jing Yang, Taowen Wang, Jinhao Duan, Mengshu Sun, Peiyan Dong, Xuan Shen, Yang Cao, Renjing Xu, Kaidi Xu, Jindong Gu, Bo Zhang, Jize Zhang, Chenhao Lin, Philip Torr, Chao Shen
Abstract
Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services. While such capabilities enable complex real-world workflows, they also introduce security risks that are difficult to capture with existing evaluations. Current agent security benchmarks often rely on manually curated tasks, provide limited coverage of emerging threats, and focus primarily on final outcomes rather than the execution processes that lead to unsafe behavior. We introduce SeClaw, a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents. Spec-driven security task synthesis enables scalable and controllable construction of security tasks from structured risk specifications, while SeClaw docker provides a standardized testbed for evaluating agent behavior under diverse safety-risk scenarios. The benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors, and supports trajectory-aware assessment of unsafe actions beyond final responses. By bridging systematic task synthesis and reproducible security evaluation, SeClaw provides a practical foundation for measuring, diagnosing, and comparing security failures in autonomous LLM agents. The code is available at https://github.com/seclaw-eval/seclaw-eval.