Context-Aware Workflow Decomposition for Automated Mobile UI Annotation Using Multimodal Large Language Models
2026-06-01 • Human-Computer Interaction
Human-Computer Interaction
AI summaryⓘ
The authors explore how to use big language models to label parts of mobile app screens automatically, which is usually hard because these screens have many small and tricky elements. Instead of doing all the labeling at once, they break the task into smaller steps with clear instructions for each type of element. Their tests show that doing the labeling in two steps works best for accuracy, while more steps find more elements but also add mistakes. This approach helps make automatic mobile UI annotation better for things like building datasets and creating smart app tools.
mobile user interfaceannotationlarge language modelsworkflow designUI elementsprecisionrecallMUIAnno datasetJSON outputprompt engineering
Authors
Athar Parvez, Muhammad Jawad Mufti, Muqaddas Gull, Omar Hammad
Abstract
Accurate mobile user interface annotation is important for UI understanding, accessibility tools, automated testing, dataset construction, and GUI agents. However, mobile screens are difficult to annotate because they often contain small, dense, nested, and visually ambiguous elements. Multimodal large language models can help automate this process, but their outputs are sensitive to prompt design and the organization of annotation tasks. This paper studies automated mobile UI annotation from a workflow design perspective, focusing on improving annotation precision. Rather than asking the model to annotate all UI elements in a single step, the task is divided into smaller context-aware stages, allowing related UI elements to be handled with clearer instructions and useful screen context. The proposed pipeline uses structured prompts, schema-constrained JSON outputs, and element-specific annotation instructions. Experiments are conducted on expert-annotated mobile UI screens from the MUIAnno dataset, using eight common UI element types: button, tab, clickable text, card, label, plain text, icon, and image. Four workflow strategies are evaluated: one-step, two-step, four-step, and eight-step annotation. Results show that the two-step workflow achieves the highest precision, while deeper decomposition improves recall but produces more false positives. Additional grouping experiments show that annotation quality depends on both workflow depth and element-class grouping. Overall, careful workflow design can make LLM-based mobile UI annotation more reliable for UI understanding, dataset construction, and GUI agent development.