Gui agents improve task success by detecting scrollable hidden areas
Chart the Invisible: Interactive Perception for Hidden-Target Exploration in GUI Agents
Software Engineering
Summary
Graphical user interfaces (GUIs) often hide important parts of the screen that need swiping to see, especially on mobile devices. The authors found that existing AI agents trying to use GUIs often click when they should swipe because they can't tell where content is hidden. To fix this, they trained a new model called Hagasu that helps these agents recognize which screen areas can be scrolled. This improved the agents' ability to complete tasks that require exploring hidden content by a significant margin.
What this means in practice
- •For mobile app developers: Enable apps to integrate smarter AI agents that better detect scrollable content and improve automated GUI task handling.
- •For software testing teams: Improve automated testing tools by equipping them with the ability to identify and explore hidden GUI components requiring swipes.
Authors
Yongxiang Hu, Ruiyang Liu, Qining He, Xuan Wang, Yangfan Zhou
Abstract
Despite rapid progress in command-line interface (CLI) tasks, recent LLM-based agents still struggle with reliable task execution through graphical user interfaces (GUIs). A key challenge is the limited content visible within a bounded viewport, especially on mobile devices, where many task-relevant targets remain hidden and can only be revealed through swipe interactions. Therefore, although most GUI agents support swipe as an action primitive, the lack of target visibility makes it difficult for them to decide when, where, and how to swipe to reveal the hidden content. Our quantitative study further shows that GUI agents incorrectly choose a click action in 47.8% of cases that require swipe exploration. To bridge this gap, this paper proposes equipping GUI agents with scrollable-region perception to support hidden-area exploration before action selection. To this end, we trained the first multimodal model for scrollable-region perception on a manually collected dataset of 667 UI screens, called Hagasu. By correctly identifying 78.5% of scrollable regions, experiments across nine GUI agents show that integrating Hagasu improves the mean end-to-end completion rate from 64.8% to 82.7% on explore-dependent tasks. To facilitate follow-up research, we release the swipe dataset and the source code of Hagasu.