GOTS: Greedy Orthogonal Token Selection for High-Resolution Vision-Language Models
2026-07-27 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors look at how to reduce the number of visual tokens in vision-language models without losing important information. Instead of judging each token alone or by pairs, they measure how different each token's information is compared to the tokens already kept. They introduce a simple method called Greedy Orthogonal Token Selection (GOTS) that picks tokens adding the most new information step-by-step. Their tests show that GOTS performs better than other methods and helps speed up processing in some cases. They also share their code publicly for others to use.
vision-language modelsvisual tokenstoken reductionorthogonalityGram determinantsubset selectiongreedy algorithmhigh-resolution encodinginference costOCRBench
Authors
Jun Ling, Tao Huang, Junzhuo Liu, Bowen Tang, Peng Wang
Abstract
Modern vision-language models (VLMs) increasingly rely on dynamic or high-resolution visual encoding, producing thousands of visual tokens that substantially increase downstream language-model inference cost. Existing token-reduction methods assess token utility through token-wise importance, query relevance, coverage, pairwise diversity, or subset-level objectives. Our key insight is to view visual token reduction through selected-span complementarity: instead of scoring a token in isolation or through pairwise relations, we assess how much of its feature is orthogonal to the span of the already retained subset. Based on this perspective, we propose Greedy Orthogonal Token Selection (GOTS), a training-free and query-agnostic method. At each step, GOTS selects the token with the largest residual energy orthogonal to the current retained span. This rule exactly maximizes the one-step augmented Gram determinant among candidate additions, giving each greedy step a precise local geometric guarantee for subset expansion. Across five high-resolution VLM backbones from the Qwen-VL and InternVL families and eleven diverse benchmarks, GOTS achieves higher average performance retention than the strongest evaluated baselines, and a controlled OCRBench study shows that it reduces model-side time-to-first-token after accounting for selection overhead. Code is available at https://github.com/newLLing/GOTS.