Misleading charts reduce web agents task success by a third
When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks
Computation and Language
Summary
Web agents are computer programs that browse and understand websites to help people. This paper finds that when these agents see misleading charts, they make bad decisions more often, dropping their task success by over 34%. The authors built a special test to compare how agents perform with clean versus misleading charts, showing that deceptive visuals cause predictable errors. Their work highlights that web agents need better ways to handle tricky charts to be safer and more reliable.
What this means in practice
- •For web developers: Test how web-based assistance tools handle misleading data visuals before deployment to improve reliability.
- •For dashboard designers: Evaluate visualizations for potential to mislead automated systems that act on dashboards.
Authors
Yisheng Li, Dongming Han, Hongkun Pan, Bingcheng Mao, Zijian Wang, Yunlong Mao, Wei Tong, Sheng Zhong
Abstract
Web agents increasingly browse, interpret, and act on web content on behalf of users. In many realistic workflows, however, the visual evidence they consume is not neutral: charts and dashboards can be misleading, shaping judgments without issuing any explicit malicious instruction. Unlike chart question answering, Web Agents treat charts as intermediate evidence whose misinterpretation can propagate into concrete downstream actions --- a reliability risk that remains underexplored. We introduce MisVisAgentBench, the first paired executable benchmark for evaluating how misleading visualizations affect Web Agent execution. It comprises 280 task instances in matched clean--misleading pairs that differ only in chart evidence. We further provide a diagnostic framework that isolates misleading-induced failures from baseline task difficulty (Zhu et al., 2025) and captures execution-level effects such as extra browsing steps and decision hesitation. Across thirteen vision-language Web Agents, replacing clean charts with misleading counterparts reduces the average task success rate by more than 34 percentage points. Moreover, these failures predominantly follow the targeted misleading action branch, suggesting that deceptive charts systematically redirect rather than randomly disrupt agent decisions. These results reveal that visualization robustness is a critical yet understudied dimension of Web Agent safety and reliability.