Interactive video worlds improve accuracy by predicting the next scene
FutureWorld: A Plug-and-Play Causal Next-State Predictor for Interactive Video World Exploration
Computer Vision and Pattern Recognition
Summary
Interactive video systems try to create scenes that change naturally when users take actions, but mistakes can build up over time, making the video blurry or inconsistent. The authors found that relying on past generated images to predict future scenes causes errors to accumulate. To solve this, they propose a method that predicts the next scene before generating it, which helps guide the video creation and reduce error growth. Their method works with different video generators and improves consistency in long interactive videos.
What this means in practice
- •For game developers: Create interactive video game environments that maintain scene consistency over long player interactions by predicting future states before rendering scenes.$Commercial implications: Enables game engines to produce more stable and consistent interactive video worlds, improving player experience and reducing graphical errors.
- •For virtual reality designers: Develop virtual reality experiences with smoother scene transitions and fewer artifacts by using predictive models for future world states.
Authors
Guangyuan Li, Kelu Yao, Shuolin Xu, Xinlei Yu, Wei Xing, Lei Zhao, Peng-Tao Jiang, Xiaogang Xu, Xiaobin Hu
Abstract
Interactive video world models aim to follow user actions while maintaining scene consistency over extended interactions. History-based methods reuse generated latents to preserve scene content, but three limitations of this reuse contribute to error recirculation. 1) The latent generated at each interaction is retained as history, potentially containing blurred details, geometric distortions, or semantic drift. 2) Although retrieved historical content matches the current scene, it can still contain errors that are carried into the next chunk. 3) New outputs then become history for later interactions, potentially reinforcing the same errors through repeated reuse. To limit this cycle, we aim to predict the future scene state from the current generated latent and the next action before generating the next interaction's video chunk. This future-state prior provides guidance for the next chunk, with the aim of reducing the errors carried forward from history. To this end, we propose a plug-and-play Causal Next-Chunk Predictor (C-NCP) with cross-model adaptability and prediction-guided modulation. We integrate C-NCP into four base generators and evaluate the augmented models across three benchmarks for interactive long-video generation and action-conditioned world modeling. Extensive experiments demonstrate reduced error accumulation and improved long-horizon consistency across diverse generation settings. The project page is at https://GuangYuanKK.github.io/futureworld.