Gui agents focus on key inputs for better task accuracy
Focused Omni-Modal GUI Agents
Computer Vision and Pattern Recognition
Summary
Graphical user interface (GUI) agents often have to deal with a lot of information at once, like screenshots, audio, video, and device data. The authors found that treating all this information equally at every step can confuse the agent because relevant details change quickly. They designed a new method called FocusGUI that helps agents pay attention only to the most important information for each action. This approach improves the agent's performance without needing extra labels or changing the main model.
What this means in practice
- •For ui automation developers: Improve automated user interface control by focusing input processing on the most relevant visual and audio signals at each interaction step.
- •For mobile app developers: Enhance app assistants that interact through multiple input types by dynamically selecting key information without retraining core models.
Authors
Fuyao Zhang, Wenwen He, Jiaming Zhang, Yurong Hao, Xuan Wang, Guowei Guan, Che Wang, Longtao Huang, Wei Yang Bryan Lim
Abstract
Graphical User Interface (GUI) agents increasingly act over streaming omni-modal observations spanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into Sequential Distraction, driving the agent into irreversible trajectory drift. We formulate the underlying requirement as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. We realize this in FocusGUI, a lightweight adapter that dynamically prioritizes multimodal channels against a step-specific decision reference. Instead of uniformly processing all inputs, it compresses the most relevant evidence into a compact latent and subtly injects it into the frozen decoder's embeddings, guiding the model's attention without altering its native parameter space. FocusGUI trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning. Across four omni-modal backbones, it raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average, and Type Match by +30.1 over the strongest existing enhancement method.