GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots
2026-06-29 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
AI summaryⓘ
The authors address the difficulty of collecting large labeled datasets for GUI agents by introducing GUICrafter, a method that learns from many unlabeled screenshots to understand interfaces without needing much human annotation. Their training happens in two steps: first, learning from lots of unlabeled data, and then fine-tuning with a small amount of carefully labeled examples. This approach helps their model perform as well or better than previous methods while using far less labeled data. Overall, their work makes building smart GUI agents more practical by reducing the reliance on expensive manual labeling.
GUI agentsvisual groundingcurriculum learningreinforcement learningunlabeled datahuman annotationcross-device generalizationfoundation modelsfine-grained GUI elementsdata efficiency
Authors
Sunqi Fan, Lingshan Chen, Runqi Yin, Qingle Liu, Yongming Rao, Meng-Hao Guo, Shi-Min Hu
Abstract
Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models. Naturally, researchers aim to extend this paradigm to the domain of GUI agents, hoping to build strong GUI agents through a similar paradigm. However, GUI agent data cannot be directly harvested from the internet, making it costly and difficult to collect at scale. As a result, current GUI agents suffer from poor cross-device generalization and limited visual grounding ability for fine-grained GUI elements. As an attempt to address data challenge in GUI agents, we propose GUICrafter, a weakly-supervised GUI agent leveraging massive unannotated screenshots to substantially reduce the reliance on expensive human annotations. GUICrafter explores a curriculum learning framework for training GUI agents through two progressive stages. First, the model learns visual grounding from large-scale unannotated screenshots and webpages, leveraging the rich contextual signals inherent in GUI interactions without human annotations. Then, in Stage 2, we leverage a small amount of high-quality data to calibrate the model via reinforcement learning. Experiments show that GUICrafter achieves competitive, or even superior, performance to advanced systems like UI-TARS while using only 0.1% of its data. Furthermore, under the same amount of annotated data, GUICrafter surpasses all previous methods such as GUI-R1. Code, data, and models are available at https://github.com/fansunqi/GUICrafter.