Memory methods improve autoregressive video generation over time
The Past Frames the Future: Memory for Autoregressive Video Generation
Computer Vision and Pattern Recognition
Summary
Generating videos step-by-step faces a problem when past details get lost after a short time, making it hard to keep things consistent like the identity of objects or changes caused by actions. This paper reviews different strategies for keeping memory during video generation, explaining how information is stored, updated, and used to improve videos over long periods. The authors also discuss how to teach these systems to remember better and how to test if memory is really being used. Their work helps create systems that generate longer, more coherent videos by remembering important past details.
What this means in practice
- •For video game developers: Design persistent world states where characters and environments maintain consistent behaviors over long gameplay through improved memory handling in generated video content.
- •For visual effects teams: Develop generative tools that maintain object identity and dynamic changes across frames for long, complex video sequences to support special effects production.
A survey. It maps existing work.
Authors
Harold Haodong Chen, Rongjin Guo, Disen Lan, Wen-Jie Shu, Hongfei Zhang, Hanzhe Hu, Shengtao Yao, Zixin Zhang, Guibin Zhang, Zhefan Rao, Jinxiu Liu, Yexin Liu, Rui Peng, Yuhao Liu, Bin Ren, Shuai Yang, Yukang Chen, Salman Khan, Ying-Cong Chen, Ser-Nam Lim, Rynson W. H. Lau, Nicu Sebe, Yu Cheng, Ming-Hsuan Yang, Qifeng Chen
Abstract
Advances in generative models have improved video fidelity, enabling long-horizon generation, interactive world modeling, and evolving visual environments. Autoregressive (AR) video generation extends visual sequences through causal rollouts. However, a fundamental bottleneck emerges: as the generated sequence expands, practical models must operate under strictly bounded context windows, storage, and computational limits. Consequently, critical historical information, e.g., entity identities, dynamic states, and intervention-induced causal changes, often leaves the active context long before its relevance diminishes. Overcoming this limitation and maintaining temporal persistence constitutes a fundamental memory problem. We present a systematic and comprehensive review of memory mechanisms in AR video generation. We formulate memory operationally as persistent historical information maintained across outer AR steps, capable of influencing future generation even after the originating evidence is no longer locally accessible. Building upon this unified framework, we organize the literature through five complementary perspectives: (I) Forms, the representational carriers of history; (II) Functions, the specific semantic and physical information requiring preservation; (III) Operations, the lifecycle of writing, reading, updating, managing, and integrating memory; (IV) Learning, the optimization of memory behaviors under closed-loop rollouts; and (V) Evaluation, the paradigms for diagnosing genuine memory capabilities. We conclude by synthesizing open challenges, including composable and resource-aware memory architectures, trustworthy state updating, self-rollout learning, and standardized evaluation. By bridging representations, mechanisms, and learning paradigms, this paper establishes a structured foundation for developing reliable, memory-conditioned video generation systems.