PhoneCLI speeds up mobile app navigation with prebuilt commands
PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents
Artificial Intelligence
Summary
Interacting with apps via vision-language models can be slow and inefficient because they have to analyze each screen every time. The authors present PhoneCLI, which explores an app beforehand to create a map of its screens and navigation paths, turning repeated navigation into quick, pre-recorded commands. This lets mobile agents perform many common tasks much faster by running these commands without needing to use the slow vision-language analysis each time. If a command doesn’t work or a new task appears, the system falls back to the original slower method, so using PhoneCLI only helps or makes no difference.
What this means in practice
- •For mobile app developers: Create faster, more reliable automation scripts for app navigation without needing app source code or APIs.
- •For automation platform providers: Integrate precompiled app navigation commands to reduce runtime costs and speed up mobile agent responses.$Commercial implications: Enables selling automated mobile agent services at lower latency and cost by reducing reliance on costly vision-language model calls.
Authors
Yangqin Jiang, Lingrui Xu, Chao Huang
Abstract
Mobile GUI agents operate through a perception--action loop: at each step they screenshot the device, invoke a vision--language model (VLM), and emit an action. It is slow, costly, and brittle, yet most of what it does is navigation---and everyday navigation is static, ordered, and endlessly repeated. We present PhoneCLI, which compiles an app's GUI navigation into callable commands, without any app-internal API, runtime instrumentation, or model training. Offline, PhoneCLI explores a target app from the outside and distills its screens, interactive elements, and navigation edges into a semantically annotated map; each screen yields one deterministic command: a replay sequence that reaches it. Online, the agent selects a command, verifies it before execution, and then executes it deterministically in sub-second time at zero VLM cost; open-ended interaction and every failure of the compiled path fall back to the embedded VLM interpreter, exactly the pure VLM agent, so compilation can only help. On AndroidLab, PhoneCLI improves the task success rate while reducing steps and token consumption, and it transfers to AndroidWorld's official M3A agent with consistent efficiency gains. What PhoneCLI compiles is the app's navigation rather than one run, so it serves new tasks, not only repeated ones.