Mobile gui agents struggle to work across different apps consistently
Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?
Artificial Intelligence
Summary
Mobile agents that control phone apps often fail to do the same task well when switching between different apps that offer similar features. The authors made a new benchmark called AnyAppBench to test how well agents can handle tasks across many apps with the same goal. They found that success in one app does not guarantee success in others, even when breaking tasks into smaller steps. This shows it is still hard to make reliable mobile assistants that work everywhere.
What this means in practice
- •For mobile app developers: Evaluate and improve mobile agents that automate user tasks across multiple Android apps offering similar functions.
- •For user interface designers: Test how mobile automation agents handle different app layouts for the same task, to inform design choices that support better agent usability.
Authors
Tien Tran, Namho Koh, Daiki E. Matsunaga, Ayush Jain, Kee Eung Kim
Abstract
Mobile GUI agents deployed in real settings must work across different applications that support the same functionality. Most existing benchmarks test each task in only one app, so a high score can mean the agent understands the task, or only that it knows that particular app. We introduce AnyAppBench, a category-controlled live Android benchmark that evaluates cross-application generalization while keeping the user goal fixed. It spans 10 functional categories, 100 task templates, and 520 task--application pairs over 52 applications. Agents run from raw instructions and with app-independent sub-goals, and a VLM judge labels every failed run under a fixed failure taxonomy whose reliability is measured by human annotation. We find that, across 13 agents, success on the original application does not transfer reliably to new applications with the same goal. Furthermore, providing high-level sub-goal decomposition produces only small, category-dependent changes that do not close the gap, and the mix of failure types changes with the target interface. Based on those insights, we believe the AnyAppBench benchmark provides an important stepping stone toward robust real-world deployment of mobile GUI agents. Our code, data and the leaderboard can be found at the project website https://anyappbench.github.io/.