Gui agents learn better step-by-step actions from their own multi-turn experience
Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents
Artificial Intelligence
Summary
Getting computers to use software by following complex instructions is hard because it needs careful thinking at every step and remembering things over time. The authors made a new way for these computer helpers, called GUI agents, to learn from their own recent actions and guidance, improving how well they follow rules and remember information during longer conversations with apps. They tested their method on two popular benchmarks and found it worked better than older methods. This helps build smarter software assistants that can interact with apps more reliably.
What this means in practice
- •For mobile app developers: Build app automation tools that can follow complex user instructions over multiple steps with improved accuracy and memory.
- •For software automation engineers: Create more reliable GUI testing agents that learn from their own interaction history to handle long interaction sequences.
Authors
Yan Zhang, Daiqing Wu, Huawen Shen, Liang Li, Gang Cao, Zhi Gong, Wei Dai, Xiaode Zhang, Can Ma, Yu Zhou
Abstract
Graphical User Interface (GUI) agents enable the fulfillment of complex user instructions through multi-turn interactions with software environments, requiring step-wise reasoning and long-horizon memory to guide actions and retain task-relevant information, respectively. Recent on-policy self-distillation (OPSD) methods have achieved strong performance on GUI grounding, a foundational subtask for GUI agents, owing to dense token-level supervision from privilege-conditioned self-teachers. However, extending existing OPSD methods to multi-turn GUI agents is hindered by self-teachers' limited privilege-following ability and insufficient privileged guidance. In this paper, we introduce GUI-SD-v2, the next version of GUI-SD, which extends OPSD from GUI grounding to multi-turn GUI interaction and addresses key limitations through a two-stage training framework. Specifically, GUI-SD-v2 first strengthens privilege following by jointly optimizing rollouts with and without privileged guidance from the same GUI states. Furthermore, it selectively distills step-specific reasoning and memory guidance through a privilege-conditioned self-teacher, supporting action decisions and the retention of task-relevant information for subsequent interactions. Extensive experiments on two representative GUI agent benchmarks, AndroidWorld and MobileWorld, show that GUI-SD-v2 compares favorably with existing OPSD baselines while consistently outperforming the evaluated state-of-the-art methods in both Pass@1 and Pass@3 success rates. Code and training data will be publicly released.