SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis
2026-07-20 • Machine Learning
Machine Learning
AI summaryⓘ
The authors created a new method called SEE to help computer programs learn how to use mobile apps better. Instead of relying on expensive human examples or repeating common actions too much, SEE builds a map of app screens and buttons by exploring efficiently. Then, it plans smart sequences of actions to try out longer and more varied tasks without going in circles. This helps the programs get better at completing tasks and adapting to new screens in real apps.
Graphical User Interface (GUI)Vision-language modelsData synthesisUser interface transition graphMulti-step trajectoriesExploration strategiesOn-policy learningTask generalizationMobile applicationsLong-horizon interactions
Authors
Zhuohang Fan, Beichen Zhang, Yuanfa Li, Changqiao Wu, Wei Liu, Jian Luan, Weigang Zhang
Abstract
Graphical User Interface (GUI) agents powered by vision-language models hold promise for automating real-world mobile tasks. However, progress is limited by the lack of high-coverage, long-horizon interaction trajectories collected from element-rich and rapidly evolving apps. Existing pipelines often rely on costly human demonstrations or on-policy framework, which tends to over-sample common flows while missing rare transitions and complex multi-step procedures. To address this problem, we propose SEE, a two-stage data synthesis framework consisting of (i) an efficient exploration stage that builds an explicit UI transition graph over screens and elements, and (ii) a graph-based synthesis stage that composes diverse multi-step trajectories via planning and controlled sampling. This design yields reproducible and explainable data generation, while explicitly preventing spurious cycles and enabling long-horizon composition. Across multiple real-world apps, SEE produces trajectories with an average length of 14.8 steps while avoiding spurious loops, and agents fine-tuned on SEE achieve improved task success and generalization to unseen screens. We will publicly release our synthesis code and dataset.