Video memory management improves streaming video understanding accuracy and efficiency
Video-MemComp: Taylor-Guided Sensitivity-Aware Memory Management for Streaming Video Understanding
Computer Vision and Pattern Recognition
Summary
Handling long video streams for AI models is hard because it needs a lot of memory and can introduce errors when compressing data. The authors created Video-MemComp, which cleverly manages memory by analyzing how sensitive different parts of the video processing are to changes. Their method carefully groups frame tokens and decides where to spend memory to keep important details. This approach works better than previous methods, improving accuracy while greatly reducing memory use.
What this means in practice
- •For video streaming platforms: Enable efficient real-time video analysis with reduced memory needs while maintaining high accuracy in understanding video content.$Commercial implications: Supports video analytics services that require streaming video comprehension with lower hardware costs and enhanced performance.
- •For ai system engineers: Design AI models for continuous video input that manage memory use based on layer sensitivity to improve efficiency without retraining.
Authors
Yonghui Niu, Yuhao Ji, Chang Nie, Jiao Zhang, Yilong Li, Junlan Feng, Ran He, Caifeng Shan, Chaoyou Fu
Abstract
The rapid growth of video data has created an urgent need for efficient video understanding systems that can handle long and continuous visual inputs, posing significant challenges for Multimodal Large Language Models (MLLMs) due to severe memory bottlenecks. Moreover, existing compression methods often overlook the explosive accumulation of approximation errors. To address this issue, we propose Video-MemComp, a training-free and sensitivity-aware context management framework for memory-constrained streaming video understanding. Our method theoretically decomposes compression error into orthogonal local approximation error and layer-wise structural sensitivity via a first-order Taylor expansion. To minimize local approximation error, we perform token aggregation in a pre-RoPE safe zone with an attention-conserving logit bias, mitigating geometric distortions caused by RoPE rotations. To manage layer-wise structural sensitivity, we dynamically allocate the KV cache budget based on offline gradient priors and online context entropy, prioritizing high-sensitivity layers. Video-MemComp outperforms or achieves performance comparable to full-cache and state-of-the-art compression methods on benchmarks including MLVU, VideoMME, and EgoSchema. Notably, it improves the overall accuracy of Qwen2.5-VL-3B from 62.3% to 64.1% while reducing the KV cache to an equivalent of 6,000 tokens per layer, saving over 90% of KV memory.