Gui agents reduce memory use with smarter image token pruning
TRACE: Trajectory-robust Admission with Evidence Ordering for Efficient GUI Agents
Computer Vision and Pattern Recognition
Summary
GUI agents take many screenshots as they interact with interfaces, but keeping all these images slows them down and uses lots of memory. The authors propose a way to pick and keep only the most useful parts of these images, making sure important areas aren’t missed even under strict limits. Their approach orders the image pieces so that the system can always reuse the best information efficiently as it works through tasks. Tests show this method works well on several benchmarks and different models.
What this means in practice
- •For mobile app developers: Improve app-driven AI assistants by reducing memory use and speeding up GUI interaction understanding under hardware limits.
- •For robotics interface engineers: Enable robots to efficiently process visual GUI inputs by selecting crucial screen areas without retraining models for every task.
Authors
Yuhao Wang, Mu Qiao, Xindong Zhang, Yunzhi Zhuge, Lei Zhang, Huchuan Lu
Abstract
GUI agents accumulate high-resolution screenshots as the trajectory unfolds, increasing inference latency and memory usage. Training-free visual token pruning can reduce this cost, but cache reuse introduces a fundamental constraint. Once tokens are discarded, the corresponding visual evidence cannot be recovered without re-encoding. Pruning therefore becomes an \textit{irreversible admission decision} that must remain useful for unknown future targets while preserving coverage of operable regions under tight budgets. To address these challenges, we propose \textbf{\method{}}, a training-free framework for \emph{\textbf{T}rajectory-\textbf{r}obust \textbf{A}dmission and \textbf{C}overage-aware \textbf{E}vidence ordering}. Specifically, we combine a query-independent layout-derived interaction prior with instruction relevance and feature novelty to rank visual evidence according to both potential future utility and diversity. Then, we reserve part of the budget for native visual tokens distributed across the screen, repairing missing spatial coverage without breaking the ordering. Together, these mechanisms produce a nested token order, allowing retained visual evidence to shrink monotonically across budgets while remaining reusable throughout the trajectory. Finally, our monotone KV contraction incrementally contracts retired frames into compact session state, avoiding repeated visual encoding or pruning. Extensive experiments across six GUI benchmarks and diverse models verify the effectiveness of our proposed \method{} under tight budgets. The source code will be released.