Towards Memory-Efficient Autoregressive Video Generation via Instance-Specific Parametric Absorption
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMultimedia
AI summaryⓘ
The authors address a problem in autoregressive video models where keeping track of all past information (KV cache) grows too large and slows down the process. Instead of simply dropping some past data, which can cause video quality issues, they propose a method called Instance-Specific Parametric Absorption (ISPA). ISPA works by converting some parts of the model to use a smaller, local version of attention and adjusting the model’s internal weights to remember the important past information. Their experiments show this method can reduce the memory needed by half without losing much video quality. This approach suggests new ways to manage memory inside streaming generative models.
autoregressive modelsstreaming video generationKey-Value cacheFull-AttentionLocal-Attentionmemory compressionparametric absorptionleast-squares optimizationlong-range dependenciesmodel weight modulation
Authors
Xiaomeng Fu, Jia Li, Yiming Hu, Yong Wang, Hayden Kwok-Hay So, Jiao Dai, Xiangxiang Chu, Jizhong Han
Abstract
Autoregressive (AR) streaming models have emerged as a powerful paradigm for long video generation. However, the linearly growing Key-Value (KV) cache poses a significant bottleneck, leading to memory overload and degraded inference throughput. A common compression method is to drop redundant KV tokens, which often breaks long-range dependencies, resulting in temporal flickering and identity loss. In this paper, we propose Instance-Specific Parametric Absorption (ISPA), a novel framework that shifts the KV cache compression from discarding to distilling. The core idea is to transit a subset of layers from Full-Attention (F-Layers) to memory-efficient Local-Attention (L-Layers) by "absorbing" historical context into the model's weights. Specifically, during a brief warmup phase, ISPA monitors the output discrepancy between global and local attention. At the transition point, we solve a closed-form least-squares problem to compute an instance-specific weight modulation that compensates for the missing history. Experiments across architectures (1.3B to 14B) demonstrate that ISPA can remove up to 50\% of the KV cache with near-lossless visual quality. We hope this perspective encourages future work to explore parametric memory consolidation beyond external token-level cache management for streaming generative models.