MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks
2026-08-24 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created MobilePA-Bench, a test system to evaluate how well mobile AI helpers can plan and use apps together on phones. Unlike older tests that either only looked at simple screen actions or offline API calls, MobilePA-Bench runs in a live sandbox that mimics real phone use with many apps and tasks. It checks if these AI helpers can break down big jobs, remember user preferences, and reuse skills effectively. Their tests show that current AI models still struggle with complex mobile tasks and unexpected errors. This benchmark helps improve mobile AI by offering a realistic way to measure and teach them.
Large Language ModelsMobile Operating SystemsTool-Centric BenchmarkInteractive SandboxSub-agent CollaborationMemory UsageSkill UsageFunction CallingReinforcement LearningMobile Agents
Authors
Yi Zhu, Xiongwei Wu, Qiyi Wang, Tingyu Qu, Jiajun Liu, Sihan Cao, Long Chen, Weigao Sun, Feida Zhu, Yiran Zhong, Steven Hoi
Abstract
As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present \textbf{MobilePA-Bench}, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents. MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning $13$ functional domains and $212$ realistic mobile tools. Beyond basic tool use, it evaluates a central planning agent along three advanced dimensions: \emph{(1)~Sub-agent Collaboration}---decomposing a complex task and delegating specialized work to capable sub-agents; \emph{(2)~Memory Usage}---recalling stored memories, user profiles, and past preferences to resolve implicit requests; and \emph{(3)~Skill Usage}---invoking pre-packaged composite skills instead of planning every step from scratch. Extensive experiments show that current frontier LLMs remain unreliable in mobile settings: performance drops sharply under strict tool ordering, permission limits, and unexpected runtime errors. By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.