Gui grounding improves accuracy and speed with smart crop ranking
RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
Computer Vision and Pattern RecognitionComputation and LanguageHuman-Computer Interaction
Summary
Understanding and interacting with graphical user interfaces (GUIs) from natural language instructions is tricky because important small or similar-looking elements are hard to spot. The authors develop a two-step method called RankGround that first picks the best area of a screenshot to focus on using a lightweight reranker, then uses an expensive model just once to find the UI element. They train this reranker carefully to tell subtle differences between similar elements. This approach makes finding the right UI parts faster and more accurate than previous methods.
What this means in practice
- •For mobile app developers: Improve automated user interface interaction tools by enabling faster and more accurate identification of screen elements from text commands.
- •For software testing teams: Enhance GUI testing systems with efficient element localization to speed up validation of user flows informed by natural language.
Authors
Liyang Fan, Xinping Bi, Yitai Li, Shuaimin Li, Hui Li, Min Yang
Abstract
Graphical User Interface (GUI) grounding is a fundamental perception task for multimodal agents, enabling them to interpret natural language instructions and interact with digital interfaces. Existing methods face a fundamental trade-off between accuracy and efficiency: direct full-image inference often fails to capture small or visually similar UI elements, while multi-crop strategies improve localization at the cost of multiple expensive Vision-Language Model (VLM) calls per query. To address this challenge, we propose RankGround, a two-stage framework that achieves accurate GUI grounding with a single VLM call per query. Central to our approach is GroundRanker, a lightweight multimodal reranker that identifies the most promising crop from a dense candidate set. Because no off-the-shelf ranking dataset is available, we construct ranking supervision data from existing grounding datasets. A strict containment criterion and boundary-aware positive augmentation improve alignment and spatial coverage in cluttered layouts. GroundRanker is then trained with a two-stage curriculum: a pointwise objective first learns coarse containment, and a listwise objective refines subtle semantic and spatial distinctions among visually similar crops. Experimental results show that RankGround consistently outperforms strong baselines while reducing computational cost. It achieves 1.4 times faster inference and improves localization accuracy by 5.5% on average over the second-best method across all backbones and screen scales, establishing a new state of the art in both efficiency and precision for GUI grounding.