DF$^3$: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created a new method called Decoder-Free Feature Forecasting (DF³) for predicting future frames in video sequences without using complex decoders. Instead, their approach works entirely in a compressed feature space (latent space) and uses special queries to pull out future frame information from an existing vision model. They also designed a lightweight way to combine motion information and feature alignment for better prediction. Their tests show that DF³ is as accurate as top methods but runs more efficiently, making it useful for robotic tasks that require fast perception and decision-making.
latent spacevideo forecastingdecoder-freevision foundation modelspatial queriesfeature forecastingmotion-aware context fusioncross-correlationrobotic perceptionstate-to-task mapping
Authors
Jiaming Chen, Guoan Xu, Aoshen Huang, Haozhuo Zhang, Yang Li, Wei Pan
Abstract
Forecasting future states from video sequences is a critical challenge for autonomous robotic systems and a fundamental objective of world modeling. Prior generative methods operating at the pixel level inevitably overemphasize task-irrelevant details, leading to prohibitive computational overhead. While latent-based approaches attempt to mitigate this by predicting features directly, the persistent reliance on heavy decoders for state-to-task mapping remains a computational bottleneck. In this work, we propose Decoder-Free Feature Forecasting (DF$^3$), a novel framework that models world evolution entirely within the latent space and directly derives task outputs, completely eliminating the need for a decoder. Specifically, DF$^3$ injects learnable spatial queries into the terminal blocks of a frozen vision foundation model to extract future state representations directly. By employing a lightweight, unified Motion-Aware Context Fusion (MACF) mechanism that seamlessly integrates coarse flow warping with fine-grained latent cross-correlation, these queries interact with historical token representations to explicitly align and forecast the feature of the next frame. Subsequently, a specialized set of task queries probes these forecasted features for the downstream task. Extensive experiments on public benchmarks and zero-shot deployment in a robotic simulator demonstrate that DF$^3$ achieves performance comparable to state-of-the-art methods while offering superior efficiency and flexibility for integrated perception and control.