Mobile app task agents struggle to complete real-world instructions reliably
APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
Artificial Intelligence
Summary
It is hard to test computer programs that can control mobile apps by following natural-language instructions because real apps are complex and unpredictable. The authors created AppSim-Bench, a set of simulated apps that are realistic but can be controlled and tested consistently. This lets different mobile app agents be compared fairly. When testing 19 different agents, the authors found that these programs still fail many tasks, especially those involving many steps or calculations. This shows there is still a long way to go before mobile GUI agents work reliably in everyday apps.
mobile GUI agentsnatural-language instructionssimulationbenchmarkreproducible evaluationdeterministic testingworkflow automationnumerical reasoningtask completion rateaction overhead
Authors
Jintian Feng, Long Chen, Xiao Yu, Jiayi Dai, Chenglong Liu, Haoru Wang, Zizhen Xue, Yuxuan Shi, Ziyang Wang, Yichen Gong
Abstract
Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these goals: simplified apps lack real-world mobile complexity, whereas live commercial apps introduce uncontrolled variation from recommendations, advertisements, accounts, and changing content. We propose AppSim-Bench, which addresses this trade-off through controllable simulated apps that preserve task-relevant interaction logic while supporting deterministic evaluation. Built through a coding-agent-assisted and human-verified workflow, it contains 557 tasks across 17 high-frequency Chinese and English apps. Its controllable backend data and outcome-based verification remove major sources of environmental stochasticity, enabling reproducible cross-model comparison. Evaluating 19 GUI agents, spanning general-purpose and GUI-specialized systems, we find that autonomous mobile execution remains far from solved. The best model completes only 50.27% of tasks, and 28.55% of tasks are not solved by any agent. Further analysis shows that failures concentrate in longer workflows, numerical reasoning tasks, and inefficient trajectories marked by high action overhead and budget exhaustion. Our project is available at https://github.com/Acrab-Agentic-Labs/AppSim.