Agentic AI audits require graded independence across three key factors

Who Audits Whom, on What Substrate, with What Evidence? An Independence-Graded Audit Protocol for Agentic AI

Artificial IntelligenceCryptography and SecurityEmerging Technologies

Summary

Auditing smart AI systems that can plan and act independently is still done using simple yes-or-no rules about how separate the auditor is from the AI. The authors argue this approach is too basic and propose measuring independence in three different ways: who controls the auditor, whether the auditor uses the same underlying AI tools as the system being checked, and whether the evidence comes from verifiable sources instead of just the system itself. They developed a method to combine these three measures and tested it, showing that traditional internal audits often miss many faults. Their approach can be used to better assure AI system safety and is connected to existing AI laws and standards.

What this means in practice

  • For ai safety teams: Implement multi-dimensional independence grading to improve reliability and fault detection in AI system audits.
  • For compliance officers: Use graded independence metrics to align AI audits with evolving legal and regulatory frameworks like the EU AI Act.

Authors

Mohamed Chahine Ghanem

Abstract

Agentic AI systems plan, invoke tools and act with limited supervision; they are now both the subject of audits and, increasingly, the auditor. Independence, the foundation of assurance,is still applied to them as a binary. We argue that it must be graded along three orthogonal axes: principal independence (who controls the auditor), substrate independence (an auditor sharing the auditee's foundation-model family, toolchain or guardrails fails with it) and evidence independence (whether evidence is attestable rather than self-reported). Each axis has precedent; the contribution is to grade all three on a single audit, aggregate them by the weakest link, and apply the same rubric when the auditor is itself an agent. We give the model a formal basis by transplanting the beta-factor model of common-cause failure from reliability engineering, a seven-step protocol whose outputs a third party can verify, a structural detectability analysis of a procurement-controls agent audited at three grades, and a Monte Carlo study of the model in which a conventional internal audit of an agent-a real audit team, a second agent, provider logsp-surfaces 5.9% of the faults it could in principle see and none at all in half the fault classes. We map the triple to the EU AI Act as amended, ISO/IEC 42006, UK public-sector risk-management guidance and audit-regulator practice.
View PDFOpen arXiv