Visual Token Coding for Video Multimodal Large Language Models
Computer Vision and Pattern Recognition
Summary
The authors introduce a method called Visual Token Coding (VTC) to reduce the number of video tokens (small pieces of video data) used by multimodal large language models that work with videos. Their approach compresses video information by predicting some frames and focusing on the differences between frames, similar to classic video compression techniques. They improve this method with dynamic adjustments for resolution and token importance, calling the improved version VTC_Dy. Tests show that VTC_Dy keeps almost all the model's performance even when using only half or a quarter of the tokens, and it can be used without needing extra training. The authors also provide their code online for others to use.
Authors
Chenxin Fang, Tao Chen, JunChao You, Jun Peng, Yiyi Zhou, Rongrong Ji
Abstract
In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), and term this new approach $VTC_{Dy}$. To validate VTC, we apply it to three MLLMs and conduct experiments on multiple video understanding benchmarks. The experimental results show that VTC$_{\mathrm{Dy}}$ achieves an average performance retention of 100.1% with a 50% token budget for Qwen3-VL, while still retaining 97.8% of the average performance when the token budget is reduced to 25%. Moreover, as a plug-and-play design, VTC requires no additional tuning of MLLMs for token coding. Our code is available at https://github.com/Msr233/VTC.