Gui agent improves task success by reusing subgoal memories
Subgoal-aware Memory Graph for GUI Agent
Artificial Intelligence
Summary
Automating tasks on mobile devices can be tricky because agents need to remember what worked before to make good decisions over many steps. The authors found that remembering every single action is too detailed, while recalling entire task sequences is too general. They created a system that remembers chunks of actions tied to specific smaller goals, called subgoals, and stores these in a graph that connects related experiences. This helps the agent reuse useful past knowledge to complete tasks better, improving its success rate in tests by over 10%.
What this means in practice
- •For mobile app developers: Build smarter app automation agents that complete complex sequences by recalling reusable subgoal experiences.
- •For customer support automation teams: Enhance virtual assistants to handle multi-step user queries on mobile platforms by leveraging structured subgoal memories.
Authors
Dezhao Luo, Daniel Beechey, Derek Yuen, Weilin Luo
Abstract
GUI agents aim to automate complex user tasks by interacting with mobile interfaces, but ef- fective decision-making in long-horizon set- tings requires reusing prior experience rather than relying solely on the environment ob- servations. Existing memory-based methods typically store experience either at the single- action level or as entire trajectories. Single- step actions are often too fine-grained to carry meaningful semantics, since the success of an individual action does not necessarily imply progress towards the overall goal. In contrast, whole trajectories are often too coarse-grained and difficult to reuse across different tasks. We propose a subgoal-aware memory graph that distils past trajectories into structured, reusable evidence at the level of subgoals. We define a subgoal as consecutive actions shar- ing a single intent, the granularity at which progress toward the task becomes meaning- ful. To reduce redundancy and support cross- task reuse, we organise subgoal experiences as nodes in a graph, where related memories can be merged, extended, and retrieved based on the environment observation. On SPA-Bench, our method improves success rate by 11.11% over the baseline.