Framefold reduces memory and improves quality in long video generation
FrameFold: Response-Guided KV Cache Consolidation for Autoregressive Long Video Generation
Computer Vision and Pattern Recognition
Summary
Generating long videos with AI requires remembering a lot of past information, which can slow down the process and use a lot of computer memory. The authors studied how existing methods struggle to keep important past details without using too much memory. Their method, FrameFold, smartly combines similar past video parts to save space while keeping the important information for generating better videos. Tests show it keeps video quality higher for longer videos compared to older methods.
What this means in practice
- •For video game developers: Generate longer and visually consistent dynamic video content with limited memory during gameplay or cutscenes.
- •For visual effects studios: Produce extended AI-generated video sequences that maintain visual quality without large memory costs in post-production.
Authors
Muyu Liu, Mingming Liu, Hong Gu, Yuyao Zhang, Qin Lu
Abstract
In autoregressive video generation, retaining all historical key-value (KV) states increases memory usage and per-step attention costs as videos grow longer. A fixed cache budget bounds these costs but raises the challenge of preserving historical content that supports consistent, high-quality generation. Existing methods control cache size through history selection or consolidation, yet deciding what to retain and how to limit the loss of useful content remains a shared challenge. Our analysis shows that spatial queries depend on different historical content and that higher current attention coverage does not necessarily yield better long-video quality. We introduce FrameFold, a training-free KV cache consolidation method that compares the content retrieved by the same recent historical queries before and after merging to guide both representative construction and merge selection. Exploiting temporal redundancy between adjacent historical entries, FrameFold merges their KV states into shared representatives while retaining complete spatial grids. It fits value-merging coefficients in closed form by matching conditional attention responses, then selects the merge with the lowest remaining response distortion. Under matched KV cache budgets, experiments on Self Forcing and Causal Forcing at 30 and 60 seconds and on LongLive at 120 seconds demonstrate improved consistency and visual quality. Continuous eight-minute Self Forcing rollouts further show less quality degradation than the baselines.