Interactive gui grounding benchmark measures revealing hidden targets
IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints
Artificial Intelligence
Summary
Most existing tests check if a computer can find something within a single screenshot, assuming everything to find is already visible. The authors point out real computer interfaces often hide targets behind scrolls, hovers, or delays, which these tests miss. They created a new challenge called IGG where computers must first interact with the interface to reveal hidden targets before selecting them. Their experiments show humans outperform AI models, mainly because AI struggles with the interaction step needed to uncover targets. They find that teaching models to do these interactions better can improve performance.
What this means in practice
- •For mobile app developers: Develop and test AI agents that navigate complex apps by interacting with interfaces to find hidden elements.
- •For assistive technology designers: Design AI helpers that can reveal and access off-screen or hidden controls to improve accessibility for users with disabilities.
Authors
Kyeongseon Kim, Jiyeon Son, Tae-Hyun Oh
Abstract
GUI grounding benchmarks ask whether an agent can localize a target that is already visible in a static screenshot. Real interfaces often violate this assumption: a target may be off-screen, appear only after a state change or hover, remain occluded, or become actionable after a delay. This leaves the ability to recover target visibility unevaluated. To measure this reveal step, we introduce Interactive GUI Grounding (IGG), a benchmark in which an agent must first make the target visible through interaction and then click it. IGG contains 601 tasks across 16 controllable GUI environments and seven visibility-constraint sub-types, with a compact action space of scrolling, moving, dragging, waiting, and clicking. In our human study, participants achieve 94.6% accuracy, while GPT-5.5 reaches 79.4% and the strongest evaluated open-source VLM reaches only 31.6%. To identify the source of this gap, we conduct two controlled interventions. Telling Qwen3-VL which interaction to perform raises its accuracy only from 30.8% to 33.9%, whereas performing the interaction for the model raises it to 81.9%. The bottleneck is executing the reveal interaction, not recognizing the target once it is exposed. We further show that this capability is learnable, demonstrating that visibility recovery can be strengthened consistently across model families.