Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models
2026-06-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors explain that current methods to reduce the number of visual tokens in vision-language models permanently discard some tokens, which can cause problems because the importance of tokens changes as the model processes information. They introduce Reroute, a new method that temporarily bypasses less important tokens instead of dropping them completely, allowing tokens to be reconsidered later. This approach maintains similar computational costs while improving the model's ability to understand images, especially for tasks requiring careful linking of words and visuals. Their tests show Reroute helps keep performance high even when many tokens are reduced.
vision-language modelsvisual tokensdecoder inferencetoken reductionattention mechanismKV-cachetoken pruninggroundingrerouteVQA (Visual Question Answering)
Authors
Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu
Abstract
Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory. Existing visual-token reduction methods largely follow a rank-and-remove paradigm: they score visual tokens, keep a compact subset, and permanently discard the rest. We show that this irreversible action is fragile because visual-token importance changes across decoder depth; tokens ranked low at one stage may become relevant in later layers, especially for grounding-sensitive queries. We propose Reroute, a training-free plug-in that replaces removal with recoverable routing. At each routing stage, selected vision tokens pass through decoder blocks, while deferred tokens bypass the stage and re-enter the candidate pool at the next routing decision. Reroute reuses existing attention-score ranking rules and stage-wise schedules, preserving the theoretical TFLOPs and KV-cache budget class of the pruning method it augments. Across FastV, PDrop, and Nüwa variants on LLaVA-1.5 and Qwen backbones, reroute improves grounding under aggressive token reduction while maintaining general VQA performance. These results suggest that VLM token reduction should not be viewed only as irreversible pruning, but also as recoverable routing. The code can be found here: https://github.com/elmma/mllm-reroute/