Frame morrow improves long video generation by predicting future needs
FrameMorrow: Future-guided Frame Selection with Prospective Tokens for Long-Horizon Video Generation
Computer Vision and Pattern Recognition
Summary
Generating long videos is hard because the computer has to remember and use lots of past frames, which gets very big and slow. The researchers found that instead of focusing only on what seems important now, the system should think about what will be important in the future. They made a tool called FrameMorrow that predicts a small summary of future needs and uses that to pick the most useful past frames. This helps make longer videos look better and stay consistent with the intended actions over time.
What this means in practice
- •For video game developers: Improve long interactive video sequences by efficiently selecting past frames to maintain visual consistency and action accuracy over extended play sessions.
- •For film and animation studios: Enhance long animated video generation workflows by integrating FrameMorrow to keep quality and coherence across many generated frames without high computation costs.
Authors
Bo Yin, Xiaobin Hu, Jiaqi Zhao, Shuicheng Yan
Abstract
Long-horizon video generation requires models to effectively leverage an increasingly long generation history. As the generated history grows, retaining all previous content becomes increasingly expensive and redundant, making effective historical selection essential. Existing approaches often determine historical relevance based on the current content. However, information relevant to the present is not necessarily useful for future generation, while seemingly less relevant history may become important later. Our key insight is that historical information should be selected according to its relevance to future information needs. Capturing these needs does not require generating the full future; instead, a compact representation of what becomes important next is sufficient to guide historical selection. Building on this insight, we propose FrameMorrow, a prospective frame selector that predicts a small set of prospective tokens representing future information needs and uses them to identify relevant information from history. FrameMorrow selects explicit historical frames rather than model-specific internal states, enabling plug-and-play integration across diverse generators, including closed-source models, with little additional inference cost. We evaluate FrameMorrow across five benchmarks and 11 generative models spanning long-video generation, interactive generation, and action-conditioned world models. Extensive experiments demonstrate consistent improvements in long-range consistency, visual quality, and action alignment across diverse generation settings.