StackTok speeds up vision language models by cutting visual tokens
StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models look at images by breaking them into many small parts called tokens, but more tokens make the model slower. The authors introduce StackTok, which smartly picks fewer visual tokens based on the question to keep important details while skipping less useful ones. It does this without needing extra training and balances between focusing on relevant parts and covering the whole image. Tests show StackTok keeps performance high while using only a fraction of the tokens, making these models faster and cheaper to run.
What this means in practice
- •For mobile app developers: Build apps using vision-language models that run faster on devices by processing fewer image tokens without retraining.
- •For cloud ai service providers: Reduce computation costs of high-resolution image understanding by dynamically selecting relevant visual tokens for vision-language model inference.$Commercial implications: Enables more cost-effective AI image services with faster response times, improving profitability of cloud AI offerings.
Authors
Zhenbin Wang, Lei Zhang, Lituan Wang, Wei Huang, Yan Wang, Zhenwei Zhang
Abstract
Increasing image resolution produces ever-longer visual-token sequences in vision-language models (VLMs), substantially raising their inference cost. To reduce this overhead without retraining, existing methods select compact token subsets that prioritize query relevance, visual coverage, or a fixed trade-off between them. The appropriate balance, however, varies across queries and token budgets: localized questions favor relevance, whereas holistic questions demand broader visual coverage. We introduce StackTok, a training-free selector that treats query relevance as the objective and visual coverage as budget-calibrated support. StackTok builds a size-indexed coverage reference from a coverage-only greedy sequence and adjusts its support target using query--vision affinity entropy. A reference-gated interleaved selection policy then switches between relevance- and coverage-oriented additions according to the current subset's support deficit. For high-resolution inputs, StackTok allocates one shared token budget across crops according to the combined marginal gain of locally nominated tokens. Evaluated with five VLMs over ten distinct image-understanding benchmarks, StackTok ranks first among training-free selectors in every tested model--budget setting. On high-resolution LLaVA-NeXT-7B, it retains 95.26% of full-token performance with only 160 of 2{,}880 (5.6%) visual tokens.