Gui agent improves task success by reusing subgoal memories

Subgoal-aware Memory Graph for GUI Agent

Artificial Intelligence

Summary

Automating tasks on mobile devices can be tricky because agents need to remember what worked before to make good decisions over many steps. The authors found that remembering every single action is too detailed, while recalling entire task sequences is too general. They created a system that remembers chunks of actions tied to specific smaller goals, called subgoals, and stores these in a graph that connects related experiences. This helps the agent reuse useful past knowledge to complete tasks better, improving its success rate in tests by over 10%.

What this means in practice

Authors

Dezhao Luo, Daniel Beechey, Derek Yuen, Weilin Luo

Abstract

GUI agents aim to automate complex user tasks by interacting with mobile interfaces, but ef- fective decision-making in long-horizon set- tings requires reusing prior experience rather than relying solely on the environment ob- servations. Existing memory-based methods typically store experience either at the single- action level or as entire trajectories. Single- step actions are often too fine-grained to carry meaningful semantics, since the success of an individual action does not necessarily imply progress towards the overall goal. In contrast, whole trajectories are often too coarse-grained and difficult to reuse across different tasks. We propose a subgoal-aware memory graph that distils past trajectories into structured, reusable evidence at the level of subgoals. We define a subgoal as consecutive actions shar- ing a single intent, the granularity at which progress toward the task becomes meaning- ful. To reduce redundancy and support cross- task reuse, we organise subgoal experiences as nodes in a graph, where related memories can be merged, extended, and retrieved based on the environment observation. On SPA-Bench, our method improves success rate by 11.11% over the baseline.