Model learns from reliable negatives to improve GUI coordinate mapping
Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding
Computer Vision and Pattern Recognition
Summary
Mapping natural language instructions to exact screen points is tough because manual labeling is costly. The authors found that focusing on how confident a model is about parts of the answer helps more than looking at the whole answer. By training only on examples where the model is confident something is wrong (negative samples), they avoid learning from bad guesses. This approach improves the accuracy of tools that help computers understand and interact with user interfaces without needing extra human labels.
What this means in practice
- •For mobile app developers: Enable apps to interpret spoken or typed instructions and precisely interact with on-screen elements without extra labeled data.
- •For software testing teams: Automatically generate screen coordinate actions from test scripts more accurately by using confidence-based models that self-improve during execution.
Authors
Yizhou Liu, Fei Tang, Yuchen Yan, Zhengxi Lu, Songqin Nong, Tao Jiang, Wenhao Xu, Wenqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen
Abstract
Graphical User Interface (GUI) grounding is essential for autonomous agents to map natural language instructions to precise screen coordinates. However, existing supervised fine-tuning and reinforcement learning methods are constrained by the high cost of annotation, creating a scalability bottleneck. In this paper, we introduce a label-free test-time training paradigm driven by two key insights: (1) confidence patterns in coordinate tokens are a better indicator than full-sequence confidence, and (2) in sparse GUI coordinate spaces, negative samples offer more reliable learning signals than potentially noisy positive ones. We first propose Confidence-Anchored Learning (CAL), which utilizes coordinate-token confidence to filter pseudo-labels and assign distance-based binary rewards. Building on this, we develop Confidence-Anchored Negative Learning (CANL), which exclusively optimizes the model using negative samples to bypass the risks of incorrect positive samples. Experimental results demonstrate that CANL-7B achieves 92.1% on ScreenSpot-V2. On more challenging ScreenSpot-Pro, CANL-7B reaches 33.8%, an 8.9% absolute improvement over the base model. Our findings establish coordinate-token confidence as a powerful alternative to manual annotations for scalable GUI agent development.