Seg2Click improves GUI visual grounding with precise clickable regions
Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding
Computer Vision and Pattern Recognition
Summary
Figuring out exactly where to click on a cluttered screen based on instructions is tricky because many spots could be correct. The authors designed Seg2Click, a system that finds all the valid places to click by predicting the exact region to interact with instead of just one point. This approach helps the system handle small and densely packed buttons better and achieves higher accuracy on common test sets. Seg2Click uses bounding box data to learn these regions and then picks a click point from the predicted area using a clear rule.
What this means in practice
- •For mobile app developers: Enhance automated GUI testing tools to accurately identify clickable areas for instructions on complex app screens without extra zoom or verification steps.
- •For automated customer support teams: Improve GUI agents that follow user instructions by precisely locating where valid clicks can be executed on user interface screenshots.
Authors
Jingchao Wang, Dingjiang Huang
Abstract
Visual grounding is a central challenge for GUI agents, requiring an instruction to be mapped to a target within a crowded screenshot. Coordinate generation with single-point supervision reduces a spatially extended target to one position, despite the presence of many valid click locations. Region-based patch scoring alleviates this ambiguity, but its spatial decisions remain tied to visual-token footprints that may span the target, neighboring controls, and background. To address these limitations, we propose Seg2Click, which reformulates GUI grounding as executable-region prediction: identifying where a click can validly execute the given instruction. Using bounding-box supervision, Seg2Click predicts an executable-region mask and computes the final click point through a deterministic rule. This formulation captures the spatial extent of valid click locations and enables finer localization beyond the visual-token grid, making it well suited to small, densely arranged controls without requiring a verifier or zoom-in pass. Experiments across five backbone configurations from two model families demonstrate the effectiveness of Seg2Click. Seg2Click-9B achieves 92.06\%, 94.10\%, and 55.22\% on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro, respectively, while matched-backbone comparisons show improvements over both coordinate generation and patch-based grounding. Code will be available at https://github.com/jcwang0602/Seg2Click.