Scaling GUI Agents with Visual State Transitions
2026-07-27 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors introduce State Transition Pretraining (STP), a new way to improve computer programs that control graphical user interfaces (GUIs). They train a model to understand how actions change what is shown on the screen by predicting both the actions from changes and the next screen state from current info. This helps the model learn better visual and action connections before it is fine-tuned for specific tasks. Their approach works better than training directly on task examples and improves across different devices and benchmarks. They also find training on both prediction tasks together is more stable and performance improves with more training data.
State Transition PretrainingGUI agentsmultimodal modelinverse dynamicsforward dynamicsvisual state transitionsfine-tuningtrajectoryAgentNetBench
Authors
Xiangyan Liu, Kaixin Li, Haonan Wang, Biao Wu, Meng Fang, Longxu Dou, Chao Du, Michael Qizhe Shieh, Tianyu Pang
Abstract
We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions). This optimization equips the model with better action-grounded visual representations and an internal world model of GUI dynamics. When subsequently fine-tuned on trajectories with task instructions, our STP-trained models consistently outperform baselines trained solely via direct trajectory fine-tuning across agent benchmarks in both desktop and mobile GUI scenarios (AgentNetBench, AndroidControl, and GUIOdyssey). Further empirical studies show that joint dynamics optimization yields stable improvements over single-objective training, and downstream performance scales steadily with the volume of transition data.