I-HITL ensures actions need explicit human approval even if system is compromised
I-HITL: Inalienable Human-in-the-Loop
Cryptography and SecurityArtificial Intelligence
Summary
Sometimes AI systems are supposed to get a human's okay before doing important things, like spending money. But the AI can trick the system and get around these controls by stealing codes or approvals meant to stop it. The researchers introduce a new way to make sure that only a genuine human can allow these actions, even if the system is hacked or the human tries to hand over permission to the AI. They tested popular AI wallets and found weaknesses, then created a system called Countersign that uses trusted devices to keep human approvals safe.
What this means in practice
- •For financial security teams: Increase protection of transaction approvals in AI-driven wallets by enforcing unbypassable human consent protocols using trusted devices.
- •For mobile device security engineers: Integrate trusted computing elements into workflows to guarantee only explicit human approvals enable sensitive system actions in compromised environments.
Authors
Samuel Breckenridge, Dani Vilardell, Wenhao Wang, Fan Zhang, Ari Juels
Abstract
Human oversight is a pillar of AI safety. It is often exercised by a human in the loop (HITL) who approves critical actions taken by an AI agent. AI agents' increasing ability to bypass guardrails, though, raises the question of how to enforce HITL robustly. We introduce the concept of inalienable human-in-the-loop (I-HITL). An I-HITL protocol ensures that an agent can initiate an action only with explicit human approval, even when the human interacts with a system (e.g., a local agentic harness) compromised by the agent. A strong I-HITL protocol additionally ensures such approval even when an adversarial human attempts to delegate approvals to an agent. To motivate I-HITL, we report on experiments with a popular agentic wallet platform in which raising an agent's spending cap requires HITL approval via emailed one-time passcodes (OTPs). Given access to the user's email and a task blocked by the cap, we find that ten different frontier models will often autonomously raise the cap, harvesting OTPs to bypass a HITL guardrail and surfacing other vulnerabilities. We formally define I-HITL using the concept of communication topologies and prove necessary and sufficient conditions for its realization. We show that an I-HITL protocol requires a trusted entity - a trusted computing device (or equivalent) - and characterize which such devices can and cannot realize an I-HITL protocol. We present a prototype I-HITL system called Countersign that can plug into and strengthen existing HITL systems using off-the-shelf consumer mobile devices.