Gui agents improve navigation by predicting masked action steps
Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Prediction
Artificial Intelligence
Summary
Navigating through software interfaces is hard for automated agents because it requires making step-by-step decisions and planning ahead. The authors propose a method called MaP that helps these agents learn by predicting missing parts of interaction sequences, unifying different navigation tasks with a consistent goal. They also introduce a way for the system to handle varied types of navigation data by routing information to specialized parts of the model. Tests on multiple benchmarks show their approach works better than just mixing tasks directly, making it more reliable for multi-task navigation.
What this means in practice
- •For software automation engineers: Improve automated UI agents to complete complex user tasks across various software.
- •For mobile app developers: Enhance app testing tools with better autonomous interaction capabilities under diverse navigation scenarios.
Authors
Yan Zhang, Pei Fu, Daiqing Wu, Huawen Shen, Ruoceng Zhang, Shaojie Zhang, Jiahui Yang, Yu Zhou, Can Ma, Zhenbo Luo, Jian Luan
Abstract
Graphical User Interface (GUI) Agents autonomously interact with software to fulfill user requests, where GUI navigation stands out as the most critical and challenging capability. Mastering this capability demands a complex synergy of step-wise decision-making, state-action alignment, and long-horizon planning. While directly mixing these corresponding navigation tasks seems intuitive to simultaneously acquire these skills, such a direct combination is severely bottlenecked by inconsistent optimization objectives and profound data heterogeneity. To overcome these barriers, we propose the MaP (stands for ``\textbf{M}asked Tr\textbf{a}jectory \textbf{P}rediction''), a unified framework that seamlessly harmonizes divergent GUI navigation tasks. By modeling multi-turn GUI interactions as a trajectory and defining training objectives through component masking and prediction, MaP shifts the optimization from task-specific marginal distributions to a consistent objective. Furthermore, to handle the data heterogeneity across multiple navigation tasks, we design a role-aware adapter learning module that dynamically routes each token to a specialized representation space. Extensive experiments on five representative GUI navigation benchmarks demonstrate that MaP effectively mitigates gradient conflicts and significantly outperforms the direct mixture training, establishing a robust paradigm for multi-task GUI navigation.