Computer use agents resist branch steering attacks with new security design
Securing Computer-Use Agents Against Branch Steering Attacks
Cryptography and SecurityArtificial Intelligence
Summary
Some AI programs that help people use computers can be tricked into making dangerous decisions by sneaky changes in website content. The authors found that existing protection methods fail because the AI must make choices based on what it sees, which attackers can influence. They created a test set to measure these attacks and a new system called COBRA that stops the trickery by strictly controlling decision branches. Their solution prevents all attacks in tests while still working well for normal users.
What this means in practice
- •For software security teams: Enhance security of AI-driven automation tools interacting with dynamic web content by preventing hidden manipulation of decision paths.
- •For web automation developers: Build robust browser-based agents that safely handle varying website data without falling prey to branch steering attacks.
Authors
Giulio Zingrillo, Hanna Foerster, Ilia Shumailov, Yiren Zhao, Robert Mullins
Abstract
Modern Computer Use Agents (CUAs) directly interact with graphical user interfaces and execute third-party web tools, exposing them to indirect prompt injection across every rendered page and tool response. While the Dual-LLM pattern is the primary system-level architecture offering formal security guarantees - using an isolated Planner LLM (P-LLM) to fix execution paths before processing untrusted inputs via a Quarantined LLM (Q-LLM) - these guarantees break down in graphical environments. Because CUA interaction is inherently dynamic, plans cannot remain data-independent; they must branch based on anticipated runtime web content - covering all possible cases the agent may encounter. This exposes agents to branch steering attacks, where an adversary crafts untrusted data to coerce a CUA down a hazardous, pre-approved branch without injecting explicit instructions. We systematically study branch steering attacks and introduce STEER-Bench (101 tasks across 9 domains), showing high attack success against both standard (94.4%) and vanilla Dual-LLM (89.5%) CUAs. We then propose COBRA, an architecture that pairs trusted branching plans with ahead-of-time capability constraints, strictly bounding the parameters and destinations each branch may execute. On STEER-Bench, COBRA reduces attack success to 0% while retaining 97% benign utility.