Contrastive learning improves locating correct elements in GUIs
ConClick: Element-Wise Contrastive Learning Improves GUI Grounding
Computer Vision and Pattern Recognition
Summary
Finding the right button or part of a computer app screen when a user gives an instruction is tricky because models often click the wrong spot that looks similar. The authors studied many models and saw they often pick the same wrong choices repeatedly. They created a new approach called ConClick that helps models learn to tell apart the correct element from look-alikes by comparing them directly. This method boosts the accuracy of picking the right spot significantly and is even better when combined with a guessing strategy called ReClick. They also provide a new dataset to help train and test better GUI-understanding models.
What this means in practice
- •For mobile app developers: Improve app assistants that follow user commands by reducing errors in identifying interface elements within complex screens.
- •For automated software testing teams: Enhance automated UI testing tools by enabling more accurate recognition of specific interface elements targeted in test scripts.
Authors
Ruotian Zhang, Yuhui Zhang, Junzhe Huang, Yu Xin, Hehe Fan, Xiaoxiao Sun
Abstract
GUI grounding, the task of locating the interface element specified by a user instruction, is a fundamental capability of GUI agents. However, existing grounding models remain error-prone, often causing downstream task failures. We systematically evaluate 10 grounding models across five datasets and identify a major error pattern: most incorrect clicks hit non-target interface elements rather than the background, with different models repeatedly selecting the same distractors. Motivated by these findings, we propose ConClick, a contrastive learning framework that teaches models to distinguish the target element from plausible distractors. We construct ConClick-Data, a dataset containing 4,292 challenging target-distractor pairs, and train the model using complementary listwise ranking and pairwise margin objectives. ConClick improves the grounding accuracy of Qwen3-VL-8B from 54.71% to 66.03% on ScreenSpot-Pro, from 83.72% to 86.28% on MMBench-GUI L2, and from 63.33% to 70.20% on OSWorld-G, outperforming prior methods built on the same or similarly sized backbones. We further introduce ReClick, an inference-time strategy that revisits promising regions and compares predictions across competing candidates, yielding an additional 2.96 percentage-point improvement in average accuracy across the three benchmarks. Together, our analysis, dataset, and methods provide both empirical insights and practical tools for building more reliable GUI agents. Data and code are available at https://github.com/zhangruotian/ConClick.