SIR: Self-improving Red-teaming for Compute Use Agents
2026-08-31 • Cryptography and Security
Cryptography and SecurityArtificial Intelligence
AI summaryⓘ
The authors studied how computer use agents (CUAs), which are AI models that control a computer by seeing the screen and using input devices, can be tricked into doing harmful actions by hidden instructions in what they read. They created an attack method called SIR that cleverly combines simple instruction tricks and learns from failures to improve its attacks. Their approach works better than previous fixed attacks and can fool several advanced CUAs without stopping the intended tasks. They also showed these attack methods can work across different AI systems without extra tuning.
Computer Use AgentsVision-Language ModelsIndirect Prompt InjectionBlack Box AttackRed TeamingOperating System ControlIterative Feedback LoopAttack TransferabilityDeterministic OracleAI Safety
Authors
Chen Xiong, Zhiyuan He, Pin-Yu Chen, Stjepan Picek, Tsung-Yi Ho
Abstract
Computer use agents (CUAs) are vision-language models that perceive a screen and act on a real operating system through mouse, keyboard, and terminal, and they are increasingly deployed to automate everyday digital tasks. Because they can be exposed to untrusted content while operating, they are vulnerable to indirect prompt injection (IPI), in which an adversary plants instructions in content the agent will read and redirects it toward actions that violate the user's intent. Existing CUA safety benchmarks evaluate fixed injections written by hand, which may underestimate the risk posed by an adaptive adversary. We present SIR, a black box IPI attack that (i) composes stealthy injections from a small library of reusable principles stated in plain language and (ii) wraps composition in an iterative feedback loop that diagnoses the victim's failed trajectories and distills the bypasses into new, named strategies that are reapplied across tasks. Unlike prior red teaming of web agents, we target CUAs at the operating system level and score attacks with a fully deterministic oracle, using checks on filesystem, service, and permission state rather than an LLM judge. On experiment, we evaluate three frontier CUAs. Composing principles with feedback raises the attack success rate over a baseline written by hand, for example from 4% to 24% on Claude Opus 4.8 and from 0% to 28% on Gemini 3.5 Flash, while the benign task still completes. Principles discovered against one model further transfer to a different architecture with no additional feedback.