GuiFlow speeds up repeated interface interactions by reusing computations
GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Graphical user interface (GUI) agents need to understand many images of a screen as they interact step-by-step, but often these images and contexts barely change, making computation slow and repetitive. The authors present GUIFlow, a way to remember and update only what changes in the screen and past interactions, instead of starting fresh each time. This method reuses past computations smartly across vision and language steps, speeding up interaction without losing accuracy. Their tests show over three times faster performance both in offline tests and real-time interactions.
What this means in practice
- •For mobile app developers: Reduce lag in apps that use AI agents to automate interactions with evolving user interfaces by reusing computation smartly.
- •For robotic process automation teams: Make software bots more efficient when interacting with graphical interfaces by incrementally updating visual and language understanding.
Authors
Ye Li, Haoran Ding, Huanan Liu, Yuan Meng, Yutian Shen, Yuansong Wang, Kangye Ji, Hanyun Cui, Hongbo Duan, Chen Tang, Jiawei Li, Shu-Tao Xia, Zhi Wang
Abstract
GUI agents built on vision-language models interact with continuously evolving interfaces, yet conventional inference repeatedly recomputes representations of largely overlapping observations and histories. This mismatch makes multi-step interaction costly, while changes in context, interface content, and action arguments prevent straightforward cache reuse. We introduce GUIFlow, a framework for full-pipeline incremental inference in multi-image GUI interaction. Our central insight is that advancing a decision need not require rebuilding all supporting representations: each interaction step can instead selectively update computational states maintained across the trajectory. To support new decisions as the context evolves, GUIFlow adapts how past computation is reused: historical KV states are position-aligned and injected into the current context; change signals guide selective updates of current-observation states in the vision encoder and language model; and earlier action responses serve as speculative drafts verified by the target model. Together, these mechanisms enable cross-step reuse throughout visual encoding, prefill, and decoding without additional training, a separate draft model, or changes to the model-visible inputs and action interface. Experiments demonstrate up to 3.39x geometric-mean speedup across offline multi-image benchmarks and up to 3.49x in online, closed-loop interaction on OSWorld, while maintaining task performance broadly comparable to the dense baseline.