Summary
It’s hard to know when and how an AI system’s authority to act should change during important tasks, especially when failures can be serious. The authors propose a new way to set rules called Runtime Assurance Contracts that control when an AI can proceed, stop, switch tasks, or ask for human help based on evidence collected as it runs. Their method differs from just using a performance score by requiring some critical checks to pass before continuing. They tested this approach in simulated coding scenarios and found it matches decisions made by human judges better than simple scoring methods. These tests show the method’s promise but don’t prove safety in real-world use.
What this means in practice
- •For autonomous system engineers: Implement policy controls that enforce mandatory safety checks during critical AI actions to prevent risky behavior.
- •For clinical decision support teams: Use runtime contracts to escalate AI decisions to human review when critical evidence criteria are not met, enhancing patient safety.
Tested on simulated data.
Abstract
Benchmarks, audits, and agent protocols describe performance, permissions, and repair, but not how observed evidence should change an agent's authority during a consequential task. We call this the assurance-transition gap. We propose a Runtime Assurance Contract (RAC), a policy-level formal schema binding autonomy boundaries, component eligibility, evidence state, transition policy, human-review capacity, and non-compensatory gates. Under RAC, soft metrics may inform routing, whereas a failed or unknown mandatory gate forces retry, switch, escalation, deferral, or stop; aggregate performance cannot authorize action. We define the contract, an evidence record, a permission rule, and five invariants, and illustrate them in clinical, industrial, and judicial failure probes. We then report a deterministic failure-injection study in agentic coding: 280 constructed cases evaluated by a gate conjunction, a score-only rule, and a restricted protocol baseline. At the published example weights and threshold, the score rule admits 80 of 100 block-required injections and all 40 review-required injections. Tuned in hindsight, it matches the conjunction on this corpus. For positive weights, a positive threshold, binary risk signals, zero-signal controls, and an injected case firing each signal alone, we show that exact agreement holds if and only if the threshold does not exceed the smallest weight. A separate set of 18 hand-authored traces checks version-pinned evidence and review transitions against simpler policy variants. In a further prospective synthetic holdout of 24 episodes, two blinded LLM judges assign identical labels to all 72 action attempts; RAC and a separately implemented full stateful baseline both match these labels. These studies test mechanisms on synthetic cases; they establish neither deployed safety nor cross-domain effectiveness.