State-space visual reasoning improves action planning from images and instructions
Resolving State-Representation Mismatch: State-Space Visual Reasoning for Open-Loop VLA Planning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Planning actions based on visual scenes and instructions is hard because current methods lose details over time or introduce errors when imagining future steps. The authors present a new approach, SSVR, that keeps the initial picture and instructions fixed and uses a compact memory that updates after each action. This helps the system plan better and faster over many steps without losing important information. Their tests show it works well in different tasks, beating earlier methods.
What this means in practice
- •For robotic system developers: Improve robot planning by preserving detailed visual context and efficiently simulating action outcomes for long sequences.
- •For game ai programmers: Enhance in-game agent navigation by using compact recurrent states to plan multiple steps from visual and textual clues more reliably and faster.
Authors
Junhao Xiao, Haoxiang Zhao, Menghao Fang, Jinkui Zhang, Jinghan Yu, Xinyu Huang, Zhiyu Wu, Kaiming Xu, Yi Chen, Youjun Bao, Zhiyuan Ma
Abstract
Despite rapid progress in vision-language-action (VLA) models, existing reasoning paradigms still face a fundamental \emph{state-representation mismatch} in open-loop planning. Given only an initial observation, models must internally simulate action-conditioned state transitions, whereas text-, pixel-, and latent-space reasoning can suffer from lossy spatial compression, error-accumulating visual generation, and bypass of intermediate latent tokens, respectively, undermining reliable long-horizon planning. We propose \textbf{State-Space Visual Reasoning} (SSVR), which decouples static visual context, language constraints, and a recurrent latent state. SSVR encodes the initial image and instruction once, then conditions each action prediction on the latent state and updates it with an action-conditioned GRU. Using Qwen2.5-VL as the backbone, SSVR achieves 99.5/99.6, 96.3/98.0, and 83.9/90.6 EM/PR on FrozenLake, Maze, and MiniBehavior, substantially outperforming prior methods. Extensive experiments support the effectiveness of recurrent state modeling for VLA open-loop planning across input transformations and transfer settings. By reusing static visual-textual context and updating a compact recurrent state, SSVR supports efficient multi-step inference, achieving up to $98.58\times$ faster Maze decoding rollouts than the evaluated baselines with the prefix cache prebuilt.