Gui world models predict better by tracking hidden state history
The GUI Is Not the State: Diagnosing State Aliasing in GUI World Models
Machine LearningArtificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Sometimes computer programs that try to predict what will happen next on a phone or computer screen get confused because they only look at the current screen. The authors found out that this can cause mistakes when the screen looks the same but the real situation behind it is different. They created a way to test for these mistakes and designed a method to remember what happened before to make better guesses about what will come next. Their approach greatly improved the accuracy of these predictions and helped computer agents act smarter in apps.
What this means in practice
- •For mobile app developers: Improve app testing tools by allowing predictive models to account for hidden states behind user interfaces.
- •For game ai programmers: Enhance non-player character decision-making in complex games by recovering hidden game state from observed screens.
Authors
Dongsheng Liu, Chao Jin, Wenkui Yang, Hejin Wang, Junwei Yang, Zeren Zhang, Ziwei Chen, Huaibo Huang, Jie Cao, Ran He
Abstract
GUI World Models (GUI-WMs) are increasingly used to predict future states for agent planning and simulation, yet most existing formulations condition only on the current GUI observation and action. We identify state aliasing, where the vis- ible interface omits transition-relevant environment state, so identical observable conditions can correspond to different valid futures. To diagnose this failure mode, we introduce StateAliasBench, a diagnostic benchmark that explicitly isolates such ambiguities via strict pairing. We further propose lightweight predictive- state recovery that infers structured state from history and augments otherwise frozen GUI-WMs through a deterministic state interface. Family-specific special- ists provide state recovery across heterogeneous state types, and multi-teacher dis- tillation consolidates them into a single unified estimator. Experiments show that existing GUI-WMs exhibit systematic failures under observation-only condition- ing, while predictive-state augmentation substantially restores state-sensitive pre- diction across evaluated WMs, preserves generative fidelity, and improves down- stream performance of GUI agents on AndroidWorld. These results suggest that reliable GUI world modeling should account not only for what is visible, but also for the hidden transition state that determines what happens next.