Search-free visual planning speeds up control with direct action prediction
Beyond Action Prediction: Learning Search-Free Planning in Latent Space
RoboticsArtificial Intelligence
Summary
Planning actions for robots or AI agents in complex visual environments usually requires slow searching through many options. This paper shows a way to skip the search by teaching the AI to predict the necessary actions directly from the current view and the goal picture. The authors introduce a method called HIJP that trains the system to make these predictions consistent and reliable by looking back at future observations and checking predictions with expected outcomes. Their approach works well on several visual control tasks, achieving high success while planning much faster than search-based methods.
What this means in practice
- •For roboticists: Plan robot movement faster by predicting actions directly from visual input without expensive search.
- •For game ai developers: Generate efficient multi-step action plans for visual game agents without costly search at runtime.
- •For industrial automation teams: Accelerate visual control systems using direct action prediction for goal-directed tasks without online optimization.$Commercial implications: Enables faster and more efficient control systems for automation equipment, improving throughput and reducing delay.
Authors
Guanchen Du, Jinhao Du, Yanpeng Sun, Shan Zhang, Na Zhao, Zechao Li
Abstract
Latent world models learn action-conditioned state transitions in a compact representation space, providing a predictive model for visual planning and control. However, planning with these models typically requires costly test-time optimization over action sequences. We propose to replace this online search with a goal-conditioned inverse model that directly predicts actions from current and goal observations. A key challenge is that action prediction alone neither ensures that goal representations encode action-relevant information nor that predicted actions are consistent with the learned latent prediction model. We introduce Hindsight Inverse JEPA Planning (HIJP), a search-free framework that addresses both aspects during training. HIJP relabels future observations along offline trajectories as goals, providing action supervision across multiple temporal horizons. By retaining gradients through the goal representations, this supervision directly shapes the visual representation. HIJP further feeds predicted actions into the latent predictor model and aligns the resulting next-state representations with encoded successor observations, enforcing action-transition consistency. At inference time, the inverse dynamics model and latent predictor models are applied alternately to generate multi-step action sequences without online search. Across four visual control tasks, HIJP achieves 86.8% overall success and a 1.9-16.5x planning speedup over search-based planning.