EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
The authors created EvoGUI, a tool to test how well computer programs understand changes in graphical user interfaces (GUIs) after actions are taken. Instead of only checking if the final goal is achieved, their method breaks down the problem into smaller questions about what happened step-by-step. They tested many models and found that even the best ones are far from perfect at this, showing there's room for improvement. Their tool can help researchers better diagnose where GUI agents struggle without needing extra labeled data.
Authors
Yaohan Yang, Minglei Shi, Borui Zhang, Jie Zhou, Jiwen Lu
Abstract
GUI agents must reason about how actions transform interface states, but end-to-end success rates entangle this ability with perception, grounding, planning, and recovery. We introduce EvoGUI, a diagnostic framework that converts normalized GUI trajectories into three complementary visual question answering probes: temporal ordering, inverse action/value prediction, and contrastive one-step successor discrimination. Their labels are derived from trajectory order and logged actions, requiring no additional task-label annotation after trajectory normalization. We instantiate EvoGUI-Bench from Mind2Web and WebLINX, yielding 3,000 instances across 120 domains, and evaluate 28 vision-language model configurations zero-shot. The strongest model reaches only 60.4 EvoGain, while model scale and GUI specialization do not reliably predict performance. These results establish EvoGUI-Bench as a scalable diagnostic complement to end-to-end GUI-agent evaluation while exposing substantial headroom in state-transition understanding. The source code is publicly available at https://github.com/Yyhhh6/EvoGUI.