Gui agents learn to act well with imperfect knowledge
Learning Reliable GUI Agents under Imperfect Priors
Machine Learning
Summary
Completing tasks on computer screens can be tricky for AI agents because each app changes over time and uses unique steps that the AI hasn’t learned before. The authors show that instead of needing perfect information, AI agents perform better when they learn to handle uncertainty and mistakes in what they know about the app. They developed a way for the AI to explore app screens, gather information automatically, and train itself to recognize when its knowledge might be wrong. Experiments show this helps the agents do more tasks accurately on different apps and devices.
What this means in practice
- •For mobile app developers: Improve automated testing tools that adapt to changing user interfaces without needing constant reprogramming.
- •For robotics system engineers: Build robots that interact with varied touchscreen devices more reliably despite updates and personalization changes.
Authors
Bo Han, Qianyi Wang, Shuai Liu, Xiong Zifan, Changqiao Wu, Yuanfa Li, Pengzhi Gao, Wei Liu, Jian Luan, Heng Qu, Yunpeng Song, Zhongmin Cai
Abstract
GUI agents built on large language and vision-language models still struggle on unseen applications and complex multi-step tasks, as completing real GUI tasks depends on app-specific, temporally volatile operational knowledge that is scarce in pretraining corpora. Retrieval-augmented execution offers a natural remedy but faces two coupled bottlenecks: knowledge at scale is hard to acquire, and self-collected priors inevitably drift from the live environment due to version updates, promotions, ads, A/B tests, and personalization. We therefore argue that GUI agents should not pursue perfect knowledge but learn to act correctly under imperfect priors, and propose our framework that couples knowledge acquisition with noise-robust utilization: a structured exploration strategy traverses interactive elements, builds a UI state-transition graph, and synthesizes (task, trajectory) pairs via a VLM without human annotation; a noise-aware training strategy, grounded in a taxonomy of real GUI drift patterns, injects five types of realistic errors into self-explored trajectories to teach the agent to assess prior reliability before acting. Experiments on physical devices and online emulator benchmarks show that our method discovers more unique screens, covers more benchmark tasks, and more effectively rejects erroneous priors while leveraging correct ones, with accuracy gains that transfer across datasets.