AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions
2026-05-25 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created AgentHijack, a test to see how well computer agents using advanced language models can handle everyday glitches like pop-ups or screen changes while working on tasks. They found that even small disruptions can make these agents perform much worse, showing they are fragile. To fix this, the authors developed AgentHijack-Agent, which helps agents better understand their environment and keep track of their actions. They tested this new system and found it works better in tricky situations.
multimodal large language modelsautonomous agentsrobustness evaluationcomputer-use agentsenvironmental corruptionsbenchmarkaction generatorgroundingbehavior summarization
Authors
Jingwei Sun, Jianing Zhu, Yuanyi Li, Tongliang Liu, Xia HU, Bo Han
Abstract
Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digital workflows. However, real-world execution environments are far from ideal: pop-ups, resolution changes, and competing applications frequently interfere with agent perception and control. We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions, where the uncertainties in dynamic environment disrupt the execution flow without direct adversarial intent. Specifically, AgentHijack introduces 9 configurable common corruptions to replicate realistic imperfect scenarios. We evaluate a variety of desktop tasks that utilize MLLM-based agents and discover that even minor instances of corruption can result in substantial performance degradation, which emphasizes the fragility of agents and underscores the necessity of robustness evaluation. Afterward, we propose AgentHijack-Agent, a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking. Extensive experiments validate its effectiveness. Our code, environment, baseline models and data are publicly available at: https://AgentHijack.github.io.