Papers for

ui automation developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Gui agents focus on key inputs for better task accuracy

Focused Omni-Modal GUI Agents

Abstract: Graphical User Interface (GUI) agents increasingly act over streaming omni-modal observations spanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into Sequential Distraction, driving the agent into irreversible trajectory drift. We formulate the underlying requirement as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. We realize this in FocusGUI, a lightweight adapter that dynamically prioritizes multimodal channels against a step-specific decision reference. Instead of uniformly processing all inputs, it compresses the most relevant evidence into a compact latent and subtly injects it into the frozen decoder's embeddings, guiding the model's attention without altering its native parameter space. FocusGUI trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning. Across four omni-modal backbones, it raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average, and Type Match by +30.1 over the strongest existing enhancement method.

Mon 19 OctComputer Vision and Pattern Recognition
The gist
Graphical user interface (GUI) agents often have to deal with a lot of information at once, like screenshots, audio, video, and device data. The authors found that treating all this information equally at every step can confuse the agent because relevant details change quickly. They designed a new method called FocusGUI that helps agents pay attention only to the most important information for each action. This approach improves the agent's performance without needing extra labels or changing the main model.
Open → 2610.26203v1