Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching
2026-08-10 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors focus on improving how computers understand and find things on screen images based on user instructions. Instead of using one big model that tries to understand and locate interface elements at once, they split the job into two steps: a language model explains the instructions in detail, and a separate model finds the right spots on the screen by comparing layout patterns. This approach avoids common mistakes where the model guesses wrong locations and does better without needing complex training. Their method significantly improves accuracy on two benchmark tests for screen element selection.
GUI groundingmultimodal large language modelsvisual groundingregression-free localizationlayout-aware modelcoordinate hallucinationinstruction parsingscreen interfaceelement selectionfine-tuning
Authors
Yuke Li, Xuehan Hou
Abstract
GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates. This task faces a persistent dual obstacle: conventional grounding models lack the semantic richness to interpret abstract instructions, while end-to-end MLLMs suffer from coordinate hallucinations caused by deficient fine-grained perception. We propose a regression-free framework where a frozen MLLM performs instruction parsing and a dedicated grounding model handles precise localization without learning any coordinate regression. A frozen MLLM first elaborates the abstract instruction into a structured visual description rich in layout cues. These descriptions are then fed to a novel Layout-Aware GUI Grounding Model, which performs regression-free localization by matching against layout-prior candidates, inherently suppressing hallucinations and avoiding expensive fine-tuning. The grounding model is trained with only Text/Icon binary labels, requiring no coordinate regression parameters. On ScreenSpot-Pro, our method achieves over 20% improvement in grounding accuracy over end-to-end systems; on Mind2Web, it raises success rate and element selection rate by more than 15%. These results demonstrate that decoupling instruction understanding from layout-aware localization effectively resolves the core challenges of GUI interaction.