Fast mobile app agents use latent action prediction to speed up tasks
GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Mobile apps can be controlled by smart agents that decide what to do by reading the screen and instructions. Usually, these agents type out each action slowly one piece at a time, which uses a lot of computer power. The authors made a new method called GUI-JEPA that doesn’t need to type actions out. Instead, it guesses the best action quickly using a special way to understand screen images, letting it work much faster and with less computer memory. Their tests show it’s nearly as accurate as slower methods but runs almost four times faster and uses much fewer settings.
What this means in practice
- •For mobile app developers: Build faster and more efficient AI agents that control mobile apps without decoding actions token by token.
- •For robotics software teams: Integrate rapid action selection methods for autonomous control interfaces on touchscreen devices.
Authors
Bingchao Wang, Yongcheng Jing, Shunyu Liu, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao
Abstract
Mobile GUI agents are typically fine-tuned 7B-scale vision-language models (VLMs) that generate each action as text. However, these VLMs decode every action token by token, and even agents that score candidate actions keep a large language model or VLM in the loop, making them slow and memory-hungry. To address these issues, we propose GUI-JEPA, to our knowledge the first decoder-free GUI agent policy built on a Joint-Embedding Predictive Architecture (JEPA), which selects actions in the latent space of a JEPA encoder pretrained on unlabelled GUI screenshots. Its action- and region-conditioned latent predictor turns the screen latent into one latent per candidate action, letting lightweight heads score all candidates against the instruction in a single batched pass, without decoding a token. We further show that the scorer relies on this latent pathway and that the predictor is trained for the decision itself: a randomly initialised predictor does just as well as one pretrained on screen dynamics. On AndroidControl, GUI-JEPA is 2.8-4.0x faster on the same GPU than a Qwen2.5-VL-7B agent we fine-tune with the same inputs, while reaching 86-92% of its in-distribution accuracy with 6% of its parameters. Code is available at https://github.com/bcwang-sjtu/GUI-JEPA