Web agents vulnerable to visual attacks that hijack browsing actions
Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution
Cryptography and SecurityArtificial IntelligenceComputation and Language
Summary
Some web agents use AI models that understand both pictures and words to interact with websites automatically. The authors found that carefully designed visual tricks can fool these agents into clicking on attacker-controlled parts of a webpage, causing unwanted actions. They built a tool called WebMirage to create these visual tricks and tested it across many websites and agent setups. Their experiments showed these attacks work much better than previous methods and can bypass some defenses.
What this means in practice
- •For web security teams: Simulate sophisticated visual adversarial attacks on vision-enabled web automation to better protect browser-based agents.
- •For software developers: Test robustness of vision-grounded browser agents by injecting targeted visual perturbations that mimic real-world webpage changes.
Authors
Wanjing Han, Levi Taiji Li, Mu Zhang, Yue Jiang, Guanhong Tao
Abstract
Modern web agents built on large vision-language models process webpages, select relevant UI elements, and translate model outputs into browser actions. Existing visual red-teaming approaches use adversarial visual content to manipulate this process. However, they primarily target model inference and do not explicitly account for structured input processing or action post-processing. Consequently, model-level success does not establish control over browser execution and cannot reliably characterize end-to-end agent robustness. To address this gap, we formulate red teaming for vision-grounded web agents as an end-to-end grounding-to-execution problem, and introduce WebMirage, a framework that crafts localized visual perturbations that cause agents to select attacker-controlled content and execute the corresponding browser action across varying webpage renderings. It uses a role-slot abstraction and webpage recomposition to capture competition among webpage elements, and dataflow analysis to align optimization with action post-processing. We evaluate WebMirage across four agent configurations and six VLM backbones on 2,250 tasks covering 13 public websites and a sandbox benchmark. WebMirage achieves an average attack success rate of 91.9%, compared with 17.4% for the strongest baseline, and remains effective against three agent-level defenses.