Video large language models face high cost from long frame sequences
Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
Computer Vision and Pattern RecognitionComputation and LanguageMultimedia
Summary
Video understanding models that combine video data and large language models can do many tasks like captioning and question answering. However, these systems use a lot of computing power and memory, especially as videos get longer. The authors surveyed methods that try to make these models more efficient by reducing the information from video frames, audio, and how the large models process the input. They also compared how well each method balances accuracy and computational cost, and pointed out areas where more work is needed.
What this means in practice
- •For mobile app developers: Design video understanding features that run with less delay and power on smartphones by using efficient inference methods from the survey.
- •For cloud service engineers: Optimize server computation for video language applications by adopting parameter and token reduction techniques to lower latency and resource use.
A survey. It maps existing work.
Authors
Killian Steunou, Yannis Tevissen, Mounîm A. El Yacoubi
Abstract
Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a textual prompt. Their strong performance on captioning, question answering, retrieval and temporal grounding comes at a computation and memory cost that grows with frame count and context length, limiting deployment in real-time, mobile and resource-constrained settings. This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count. We analyze bottlenecks across frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding. We organize methods by the pipeline stage at which they act, covering VideoLLMs developed since late 2022 together with earlier frame-sampling and vision-encoder mechanisms that remain components of current pipelines. We assemble literature-reported accuracy--cost comparisons under shared host models and input protocols wherever available, distinguish them from heterogeneous cross-paper evidence, and identify gaps in audiovisual efficiency and standardized evaluation. We maintain a repository at https://github.com/momentslab/awesome-efficient-videollm.