AI agent risk certification requires balancing zero audit responses
Clean Sheets and Checkpoints: Evidence Complexity for Certifying Sequential Agent Risk
Machine LearningArtificial Intelligence
Summary
Testing AI agents for risky behavior often involves checking success or failure at many steps. The authors study how many failed checks you need to be confident that an AI’s chance of success is very low, especially when failures may wipe previous progress. They find that the number of failed checks needed depends on how many milestone groups are actively risky and how attempts are budgeted. The work explains how to divide testing effort across these milestones to best certify safety.
What this means in practice
- •For ai safety engineers: Design audits for AI agents by allocating test efforts to critical milestones to certify low risk with optimal use of failure data.
- •For security testing teams: Develop strategies for evaluating complex systems with sequential checkpoints, improving confidence in failure bounds.
A theory result. No direct application yet.
Authors
Rohith Yanambaka Venkata, Don McBride, Tiago Sequeira, Md. Al Amin
Abstract
Dangerous-capability evaluations often test an AI agent from prepared states at ordered milestones. A clean sheet records no successes. What it certifies about a campaign of $B$ attempts depends on what a failed attempt erases. We model the task as a chain of $k$ milestones divided into checkpoint segments and ask how many zero audit responses certify, at level $α$, a completion probability of at most $ε$. For segments chosen before the audit, counting possible completion windows gives a certificate for any specified budget. The bound also covers adaptive deployment processes that follow the chain's transitions, provided their success probabilities after every complete history are capped by the corresponding audit success probabilities. The audit does not test this transport assumption. In rare-risk phase interiors, each segment is either rarely completed within the budget (active) or nearly always completed (free). The minimum expected number of zero responses is asymptotic to $\log(1/α)\,(B^a/(a!\,ε))^{1/K_A}$, where the $a$ active segments contain $K_A$ milestones and are those at least as long as a budget-dependent threshold. The certificate attains this constant. Every asymptotically optimal evaluator must divide its expected response count equally among active milestones in the limit, with a vanishing share elsewhere. Where a single segment passes from active to free, each of its milestones receives a smaller limiting share than an active milestone; a crossover-aware certificate attains the optimal constant. For selected segments with known or unqueried prefix steps, counting failed cycles can sharpen the certificate.