Latent memory improves frozen gui agents by learning where to fix errors
ScopeMem: Latent Memory That Learns Where to Revise Frozen GUI Agents
Artificial Intelligence
Summary
Making computers control graphical apps is tricky, especially when you want them to remember the past steps. The authors found that a common approach fixes mistakes but also breaks some right answers unknowingly. They created a new method called ScopeMem that better knows when to fix errors without messing up good decisions. This method helps frozen vision-language models act smarter without retraining the whole system.
What this means in practice
- •For software developers: Build GUI automation tools that improve accuracy by selectively correcting agent errors without breaking correct actions.
- •For mobile app developers: Create smarter app control assistants that adjust actions on the fly while preserving correct user interface interactions.
Authors
Zhicheng Zheng, Cheng Zhang, Shengping Yu, Tianxiang Zhao
Abstract
Latent memory is gaining ground as a lightweight history supplement for GUI agents. It supplies history to a frozen vision-language model via prefix vectors trained with step-wise cross-entropy (CE). While resulting gains are naturally attributed to encoded history, our controlled experiments show they primarily stem from action revision. The prefix learns to alter the model's default actions. However, this revision is unscoped because step-wise CE pushes outputs toward reference actions at every step without explicitly preserving correct decisions. Consequently, it repairs errors but also breaks 8-13% of initially correct answers. Overall accuracy metrics hide these losses. We therefore propose ScopeMem, a latent-memory method for scoped revision. Its encoder conditions prefix generation on the model's initial answer and is trained to correct errors while preserving correct decisions. On the controlled ablation set, ScopeMem reduces the share of broken correct decisions to 5.6% while repairing more errors than any evaluated CE-trained prefix. On AndroidControl, GUI-Odyssey, and zero-shot AITW, ScopeMem outperforms all evaluated prompting and latent-memory baselines, enabling a frozen Qwen3-VL-8B to match or surpass GUI-specialized agents and Qwen3-VL-32B.