Video large language models get faster without retraining costs

Training-Free Inference Acceleration for Video Large Language Models: Methods, Trade-offs, and Evaluation

Computer Vision and Pattern Recognition

Summary

Processing long videos with large language models is very slow and uses a lot of computer memory. The authors review different ways to speed up these models during their use without needing extra training. They show how changing parts of the model’s steps affects both speed and accuracy, and why some improvements don't always make the whole process faster. They also compare past methods and run their own tests to find the best balance between speed and performance.

What this means in practice

A survey. It maps existing work.

Authors

Yuqi Li, Chuanyue Yan, Neil Rodrigues, Xianglong Wang, Xiaoqin Feng, Junming Liu, Xi Xiao, Zhongwei Cheng, Wei Li, Lin Chen, Yingli Tian

Abstract

Video Large Language Models (VLMs) have demonstrated strong capabilities in video understanding and reasoning, but processing long visual sequences incurs substantial computational, latency, and memory overhead. This survey examines training-free inference acceleration, focusing on reducing inference costs while preserving task-relevant spatiotemporal evidence. We organize existing methods by their intervention stages in the inference pipeline, covering input, visual encoding, projector, prefill, and decoding. Post-training quantization (PTQ) is discussed separately as a cross-stage extension with varying calibration requirements. We analyze how intervention position affects evidence preservation and computational savings, and why local efficiency improvements may not translate into end-to-end acceleration. We complement this analysis with quantitative evidence from prior studies and our own experiments under a shared evaluation setting to assess accuracy-latency trade-offs.