PrecogUI improves user interface agents by predicting problems early
PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval
Artificial Intelligence
Summary
Graphical user interface agents usually react to changes after they happen, which can cause problems over long tasks when unexpected issues arise. This paper introduces PrecogUI, a system that helps these agents think ahead by predicting possible problems before they occur. It learns from past successes and failures, simulates future screen layouts after potential actions, and chooses the best steps to avoid errors. Tests show that PrecogUI works better than current methods in challenging scenarios with many disturbances. This approach helps make automated UI agents more reliable for complex tasks.
What this means in practice
- •For mobile app developers: Create user interface agents that anticipate and avoid errors during complex, long workflows with frequent unexpected changes.
- •For software testing teams: Automate detection and prevention of UI issues by simulating future behaviors and learning from past test cases with anomalies.
Authors
Bin Kang, Jiarui Ouyang, Li Jiang, Bin Chen, Zhuotao Tian
Abstract
Existing reactive Graphical User Interface (GUI) agents often fail in long-horizon, dynamic scenarios, where unexpected disturbances trigger attention-diverting and cascading failures. To address this, we propose PrecogUI, a pre-cognitive architecture that shifts the paradigm from reactive execution to proactive decision-making. Specifically, we design a Proactive Experience Pool (PEP), which caches recurring anomaly and success patterns as "state-action-result" tuples in a dual-memory repository. Furthermore, we introduce a Proactive Simulation Executor (PSE) that learns to forecast the next symbolic UI layout given a candidate action, enabling early anomaly avoidance and ranking candidate actions by predicted reliability. Finally, a Pre-cognitive Execution Controller (PEC) fuses these priors and predictions, prioritizes handling of foreseen anomalies, and ensures execution robustness through a closed-loop error correction mechanism. For robust evaluation, we develop AutoTraj, an automatic data-generation engine, to construct InterfereBench, a benchmark for long-horizon tasks with strong disturbances. Experiments demonstrate that PrecogUI surpasses state-of-the-art methods on InterfereBench while maintaining competitive performance on public benchmarks. The code will be publicly available.