Runtime controls coverage measured for llm agent actions
guardscope: Measuring Which Actions Runtime Controls Check in LLM Agent Systems
Artificial Intelligence
Summary
LLM agents use multiple ways to interact with outside systems, but security checks can only see some of these actions. The authors created guardscope, a tool that measures which agent actions a security control actually monitors by comparing reported checks with a low-level record of all actions. This helps developers know exactly what their controls watch for and what might slip through. The authors also tested guardscope with real tools and multiple agent setups.
What this means in practice
- •For security engineers: Verify and quantify the actual coverage of runtime controls on agent actions in complex deployment setups.
- •For devops teams: Design and deploy LLM agent governance architectures with empirically measured security control effectiveness.
Authors
Kentaroh Toyoda, Alex Leung, Bhuvaneswari U
Abstract
Large-language model (LLM) agents execute actions on external systems through tool calls, and enterprises deploy runtime controls, such as agent gateways, to check this behavior in real time. However, a control can only evaluate the actions that cross the point where it is placed, and an agent can act through more paths than the tool-call channel; no existing work measures this coverage against kernel-level ground truth. In this paper, we present guardscope, a framework that makes this coverage measurable. guardscope works in two stages: a static stage that predicts which of an agent's 15 action paths a control can check from its source, and an empirical stage that runs a fixed workload on a deployment that realizes one of eight topology designs and classifies every executed action as checked, bypassed, or prevented by reconciling the control's decision records against kernel-level ground truth. guardscope makes a control's coverage claim verifiable: a runtime control developer can publish which paths its product inspects under a kernel-verified protocol. We conducted experiments with two agent harnesses and real runtime controls, including Microsoft's agent governance toolkit and agentgateway. guardscope is available at https://github.com/VulcanLab/guardscope.