Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models
Computer Vision and Pattern Recognition
Summary
The authors address the problem that video vision-language models (VLMs) are costly to update and run because videos produce many visual data points (tokens). They propose Token-Budget Distillation (TBD), a method that keeps the main model fixed and trains small adapter modules while compressing the video tokens to save computation. To avoid losing important information during compression, their method uses a teacher-student system where the full data teacher guides the compressed student model to maintain accuracy. The authors test TBD on multiple models and datasets, showing it performs better than simple compression methods, keeping nearly all the original accuracy even with heavy token reduction.
Authors
Xiaoyang Guo, Guoping Luo, Jusheng Zhang, Keze Wang, Wenhao Wang
Abstract
Adapting video vision-language models (VLMs) is computationally expensive because video inputs produce a large number of visual tokens, making both fine-tuning and inference costly. Although visual token compression can reduce this overhead, direct adaptation on compressed inputs often causes semantic drift and noticeable performance degradation. We present Token-Budget Distillation (TBD), a parameter-efficient fine-tuning framework for adapting video VLMs under a fixed token budget. TBD freezes the pretrained backbone, updates only LoRA adapters, and integrates FlashVID-based visual token compression into the video pathway. To preserve full-token semantics under compression, TBD employs a dual-path teacher-student design, where a full-token teacher provides stable supervision and a compressed student is optimized with task loss, answer-region KL distillation, GT-anchored margin distillation, and reliability-aware KD control. This design enables the student to recover the semantic behavior of the full-token model while remaining efficient under aggressive token reduction. We evaluate TBD on three video VLM backbones, including LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct, across four video understanding benchmarks. TBD consistently outperforms compression-only baselines under both moderate and aggressive compression. On LLaVA-Video at retention ratio R = 10 percent, TBD preserves 97.0 percent of the Vanilla model's average accuracy; on LLaVA-OneVision at R = 10 percent, it achieves an average score of 58.4 and matches 100.0 percent relative accuracy.