Small gui model action type input methods compared in android grounding tasks
Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type?
Machine LearningComputer Vision and Pattern Recognition
Summary
The paper looks at how a small computer program that controls a phone screen should be told what kind of action to perform, like tapping or swiping. The authors tested five different ways to give this action information to the program and found some methods improve accuracy more than others. However, much of the improvement depends on specific preprocessing choices, and wrong action types at run-time usually cause failures. The study focuses on one system and data type, so it’s unclear if these findings hold for other setups.
What this means in practice
- •For mobile app developers: Improve accuracy of small models that decide actions in mobile graphical interfaces by choosing effective ways to feed action type information.
- •For human-computer interaction engineers: Design better interaction agents for touchscreen devices by selecting the most robust method of encoding action types during model training.
Tested on one dataset.
Authors
Aadi Chauhan, Arthur Ilyasov
Abstract
A GUI agent decides which action to take and where to take it; we ask how a small grounding model should receive the action type. Fine-tuning Qwen2-VL-2B with LoRA on Android in the Wild, we compare a flat baseline with five ways of supplying the type under matched data, compute, and decoding: an auxiliary loss, a hard-routed action word, an additive learned embedding, a prepended learned token, and the type written into the prompt. With five seeds, an episode-clustered bootstrap, and seed-level paired tests, the ranking on a mixed stream is clear: the auxiliary loss, the additive embedding, and the prompt word each gain five to seven hit@0.10 points over the baseline, while hard routing and the prepended token are not distinguishable from it. Much of that gain is protection from a preprocessing choice of ours rather than a spatial prior. Our serializer clamps the off-screen touch point AITW records for type events to the origin; that class degrades the baseline's click grounding, and removing it lifts the baseline by nearly seven points, after which no mechanism's hit rate beats it and the intervals exclude a two-point effect, though the auxiliary loss still shortens the average miss; on a stream of taps and swipes none helps. Whether this generalizes beyond one serialization is open. For deployment, the pipeline's margin over the baseline with predicted rather than gold types is not established (+0.016, 95% interval [-0.017, +0.052]), and a wrong type collapses every model conditioned at inference. The prepended token does not help at the shared learning rate, where its rows barely move from initialization; trained ten times faster it reaches the level of the other three, with a margin three seeds do not establish. We also document a silent failure: injecting conditioning through inputs_embeds makes Qwen2-VL fall back to 1-D positions for image tokens, costing nine points.