Personal gui agents improve decisions by activating useful past experience
Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents
Computer Vision and Pattern Recognition
Summary
Computers that help you navigate software often look at your past actions to guess what you want to do next. This paper shows that just finding relevant past experiences isn't enough because what worked before might not apply to the current screen or step. The authors introduce a method called ExpActivator that picks only the parts of past experiences relevant to the exact situation, helping the agent make better decisions. This approach improves performance while using less past history, making personal GUI agents smarter and more efficient.
What this means in practice
- •For software developers: Enable smarter UI assistants that use only relevant past user actions to better guide interactions and decide when to suggest routines.
- •For enterprise software teams: Improve enterprise automation tools by activating historical workflow steps exactly when they apply to the current screen and timing.
Authors
Fuyao Zhang, Xuan Wang, Zherui Li, Jiaming Zhang, Longtao Huang, Wei Yang Bryan Lim
Abstract
Personal Graphical User Interface (GUI) agents rely on interaction history to infer what a user wants from ambiguous instructions and to anticipate recurring routines. Existing approaches retrieve task-relevant history and append it to the policy's context, implicitly assuming that experience relevant to a task remains useful for each decision within it. We find that this help is largely spent at the first decision: retrieved history strongly improves the opening step of an episode, yet provides little sustained benefit over the remaining 90\% of steps, and offers weak guidance on whether a proactive suggestion is warranted. A relevant record may tell the agent where to begin, but not which past action applies to the current screen or whether a routine is due now. The underlying issue is that relevance does not imply applicability}: relevance is determined at the task level, whereas applicability depends on the situation at decision time. We therefore recast personalization as experience activation and introduce ExpActivator, a training-free framework that activates only the experience applicable to the current situation. During execution, ExpActivator matches each new screen to historical states in the frozen GUI backbone's latent space and supplies the corresponding action as a reference. Before execution, it activates a recurring intent only when the current time and scenario provide sufficient support, and otherwise abstains. Across four GUI backbones, ExpActivator improves within-trajectory step success by 28\% on average, achieves the best personalized execution on every backbone while using about one-fifth as many history tokens, and reaches approximately 2.3$\times$ the Matthews correlation coefficient of the strongest proactive baseline. Experience pays where it is activated, not where it is appended.