Mobile gui agents evaluated with realistic and repeatable simulated apps
APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
Artificial Intelligence
Summary
It is hard to test phone app assistants because real apps change often and simplified apps don't feel real. The researchers created a set of simulated apps that behave like real ones but can be tested the same way every time. They made 557 tasks from popular apps and measured how well 19 different agents could do them. The best assistant only finished about half the tasks, showing there is more work to do. Most mistakes happened in harder tasks, like long sequences or math problems.
What this means in practice
- •For mobile app developers: Use realistic simulated apps to test and improve mobile app automation agents without unpredictability from real app changes.
- •For qa automation teams: Evaluate mobile GUI agents with consistent, repeatable tests that cover complex user tasks across multiple apps.
Authors
Jintian Feng, Long Chen, Xiao Yu, Jiayi Dai, Chenglong Liu, Haoru Wang, Zizhen Xue, Yuxuan Shi, Ziyang Wang, Yichen Gong
Abstract
Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these goals: simplified apps lack real-world mobile complexity, whereas live commercial apps introduce uncontrolled variation from recommendations, advertisements, accounts, and changing content. We propose AppSim-Bench, which addresses this trade-off through controllable simulated apps that preserve task-relevant interaction logic while supporting deterministic evaluation. Built through a coding-agent-assisted and human-verified workflow, it contains 557 tasks across 17 high-frequency Chinese and English apps. Its controllable backend data and outcome-based verification remove major sources of environmental stochasticity, enabling reproducible cross-model comparison. Evaluating 19 GUI agents, spanning general-purpose and GUI-specialized systems, we find that autonomous mobile execution remains far from solved. The best model completes only 50.27% of tasks, and 28.55% of tasks are not solved by any agent. Further analysis shows that failures concentrate in longer workflows, numerical reasoning tasks, and inefficient trajectories marked by high action overhead and budget exhaustion. Our project is available at https://github.com/Acrab-Agentic-Labs/AppSim.