Visual state encoder advances unified prediction instruction and inspection
ALPS: Visual State Encoders for World Models
Computer Vision and Pattern Recognition
Summary
Understanding what robots see and plan is hard because visual data, language, and physical states are often handled separately. The authors created ALPS, a new way to turn images and text into a combined understanding that helps predict actions, follow instructions, and check what's happening visually. ALPS was trained to keep track of physical information, understand native language, and recreate images all in one system. It performs better than many existing methods in tests involving robot tasks and lets users directly inspect predicted future visuals.
What this means in practice
- •For robotics developers: Improve robot control systems by using ALPS tokens for better visual understanding combined with language and physical state cues.
- •For game ai developers: Enable game AI to predict future visual game states with interpretable tokens that can be decoded back into images.
Authors
Yanqing Liu, Letian Zhang, Chengzhi Liu, Zijun Wang, Jieru Mei, Hongliang Fei, Yuyin Zhou, Cihang Xie
Abstract
World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models.