AnchorGUI improves success in navigating apps by smarter memory use
AnchorGUI: Asymmetric Memory for Dual-Scale Learning in GUI Navigation
Computer Vision and Pattern Recognition
Summary
Navigating apps automatically is hard because computers struggle to learn from lots of screen images and mistakes. The authors identified that it helps to keep simple text summaries for expected steps but save screenshots when something unexpected happens, so errors can be spotted and fixed better. They created AnchorGUI, which uses a special memory system to focus on mistakes immediately and learn better from past trials. This approach improves success in app navigation while using fewer computing resources.
What this means in practice
- •For mobile app developers: Automate app testing by detecting and correcting UI navigation errors more efficiently using visual evidence and prediction errors.
- •For software quality assurance teams: Improve automated GUI testing by focusing analysis on navigation failures, reducing resources needed for identifying root causes.
Authors
Shengjie Jin, Zelong Sun, Hengbo Xu, Yanbiao Ma, Zhiwu Lu
Abstract
Vision-Language Models (VLMs) enable autonomous GUI navigation, but agents still struggle to process and learn from dense, continuous visual histories. This bottleneck hinders both immediate error correction within a single episode (intra-trial) and experience distillation across multiple attempts (cross-trial). We trace these challenges to an empirical informational asymmetry in GUI navigation: while expected transitions can often be compressed into lightweight textual summaries, unexpected outcomes benefit from preserved screenshots as causal evidence for accurate diagnosis. Building on this insight, we propose AnchorGUI, a unified framework driven by the Cognitive State Anchor (CSA). The CSA acts as a per-step primitive that actively compares expected and observed transitions, converting passive multimodal trajectories into explicit prediction-error signals. These signals orchestrate a dual-scale learning mechanism via an asymmetric memory. For intra-trial correction, a sliding window selectively retains visual evidence for detected mismatches, providing immediate, visually-grounded feedback. For cross-trial distillation, this asymmetric memory focuses the computationally expensive credit assignment search space on likely failure steps. Experiments across four benchmarks validate the effectiveness of our approach. On AndroidWorld, AnchorGUI achieves a 57.3% success rate with a $2.4\times$ token reduction per step. Furthermore, cross-trial distillation reaches 69.2% success (+11.9% gain), significantly outperforming standard reflection methods while maintaining sub-linear context scaling.