Online reinforcement learning improves ios app navigation agents
Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps
Artificial Intelligence
Summary
Navigating apps on smartphones using AI is hard because big models are slow and smaller models often fail when things don’t go as planned. The authors designed a system that trains AI agents to learn by interacting directly with real iOS apps. They split the AI into two parts: one that plans steps and another that understands the app screens. They also use a smart feedback system to help the AI learn better at each step and introduced a new way to estimate values that makes learning more stable. Their approach trains agents that work better and faster than previous methods, even when tested on other devices and operating systems.
What this means in practice
- •For mobile app developers: Improve automated testing and navigation of live iOS apps through agents trained with real-time user interface interactions.
- •For software quality assurance teams: Accelerate UI automation workflows by using reinforcement learned agents capable of recovering from errors and generalizing across platforms.
- •For robotic process automation providers: Enable robust UI automation solutions that dynamically adapt by training agents online on live apps for error recovery and cross-platform use.$Commercial implications: This paper enables development of advanced automation products that function reliably on iOS and Android by training agents online.
Authors
Peitong Duan, Khashayar Gatmiry, Jeffrey Nichols, Samy Bengio, Emmanuel Abbe
Abstract
While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation. Furthermore, we introduce GiGPPO, a novel neighbor-conditioned value estimation method that improves critic stability by leveraging reward signals from visually similar states across concurrent trajectories. To support this online training paradigm at scale, we introduce an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices. Evaluations on rigorous benchmarks, including UINavBench, iOSWorld, and AndroidWorld, demonstrate that our online RL trained agents substantially outperform both SFT and terminal-only RL baselines while accelerating training convergence. Notably, our 8B RL agents achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments, while outperforming Gemini-3.5-Flash and tying with Gemini-3.1-Pro on real iOS applications.