AgentAudit evaluates trustworthiness of AI agents across all steps
AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents
Artificial Intelligence
Summary
AI agents perform complex tasks through multiple steps like planning, using tools, and reasoning, but most tests only check if they finish the job, not how trustworthy each step is. The authors created AgentAudit, a tool that watches every step in how an AI agent works and checks if it behaves correctly, safely, and as intended. This tool can work with any AI that uses large language models without changing the AI itself. By testing several popular AI models, they found differences in how trustworthy these agents are, even when task completion seemed similar.
What this means in practice
- •For ai deployment engineers: Identify precisely which step in an AI agent’s process causes failures or unsafe behavior during deployment.
- •For software security teams: Evaluate the security and execution integrity of AI agents to prevent misuse or harmful outcomes.
Authors
Shrey Nag, Sachita, Abhishek Kumar Singh, Lipi Goel, Rajeshwar Singh Janwar
Abstract
Existing evaluation frameworks mostly assess only one part of AI agents, such as task completion (AgentBench) or security robustness (AgentDojo, ASB), rather than the complete pipeline of planning, tool selection, tool execution, memory and reasoning. Failures can occur at any stage, yet existing benchmarks rarely identify their precise source. AgentAudit evaluates the entire execution trace across ten capability, grounding, security and behavioural dimensions, namely instruction integrity, planner, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security and execution integrity, combined with behavioural classification and failure attribution to pinpoint the exact stage responsible for an observed failure. AgentAudit can evaluate any LLM-based AI agent, since it attaches to the agent instead of replacing it. It reads only the recorded execution trace and does not interfere with how the agent runs, so it places no constraint on the agent's internal implementation. We evaluate five language models (OpenAI GPT-5, Claude Sonnet 5, Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash) across nine capability and adversarial tasks. Claude Sonnet 5 and GPT-5 obtain the highest mean Composite Trust Scores (95.1 and 80.6 out of 100, respectively), while Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash trail substantially (57.6, 45.7 and 22.6). All traces were scored by a single fixed judge model, which was itself one of the evaluated models, a limitation discussed in Section VII.E. More importantly, models with similar task-completion behaviour can diverge sharply in trustworthiness, as several non-frontier models are repeatedly classified Unsafe_Compliance on adversarial tasks rather than merely failing them, a distinction that pass/fail benchmarks cannot surface.