Visual memory improves long video consistency across multiple shots
REEL: Learning to Retrieve, Evaluate, and Evolve Visual Memory for Any Long-Horizon Multi-Shot Video Generation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Generating long videos that look consistent and follow a story is a hard problem. The authors show a new system called REEL that acts like a smart planner for managing visual memory across multiple video clips. REEL learns when to retrieve past images, fill in missing parts, and update what it remembers to help video generators keep characters and scenes consistent over time. It works with different video-generation tools by using adapters and is trained with special datasets and reinforcement learning. Tests show REEL helps videos keep identities and changes more consistently between clips.
What this means in practice
- •For video game developers: Create longer in-game cutscenes with better character and scene consistency across multiple shots.
- •For animation studios: Improve multi-shot animated sequences by automatically managing visual references for consistent character appearance and scene changes.
Authors
Hang Xu, Guohui Zhang, Xiaoxiao Ma, Hu Yu, Jie Huang, Jiayin Xiao, Lin Song, Haoyang Huang, Nan Duan, Feng Zhao
Abstract
Memory-based long-video generation aims to combine user control with long-range visual consistency. We organize this process into three functional components: a planner, an adapter, and a generator. Representative multi-shot systems couple memory to generator-specific interfaces or use predefined agent workflows, rather than independently training a memory policy for cross-shot objectives. We introduce REEL, a general-purpose visual-memory planner. Through a unified tool interface, REEL learns to retrieve references, create missing visual evidence, and update persistent memory. Backend-specific adapters translate its evidence plans into supported conditioning inputs, allowing the same policy to guide different black-box video generators. To train REEL, we construct two high-quality datasets, REEL-SFT-4k and REEL-RL-1k, covering diverse interaction patterns and prompt types. Training proceeds in three stages: supervised fine-tuning, single-shot Group Relative Policy Optimization (GRPO), and multi-shot GRPO. Multi-shot GRPO carries policy-generated memory across consecutive shots, so later outcomes guide earlier reference selection and memory updates. REEL-Bench tests identity preservation and intended state changes under text-only requests and timed reference-image uploads. On JoyAI-Echo~1.5 and MiniMax-H3, REEL improves cross-shot consistency on ST-Bench by 15.1\% and 20.7\%, and Core-100 on our proposed REEL-Bench by 2.41 and 8.32 points, respectively, over the no-memory baseline.