CatchUp efficiently reduces visual tokens in large vision models
CatchUp: Depth-Aware Re-Admission of Visual Tokens for Efficient Large Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Large vision-language models look at many parts of an image to understand it, which takes a lot of computing power. Current methods cut down the parts they look at but don’t always reconsider skipped parts fully later, causing missing detail. The authors found a way to reintroduce those skipped parts with updated information during processing, helping keep good results with less work. Their method, CatchUp, picks which image parts to process step-by-step and improves speed while keeping model accuracy high.
What this means in practice
- •For mobile app developers: Build vision-language features that run faster and require less memory by selectively processing image parts while preserving accuracy.
- •For autonomous vehicle software teams: Improve real-time visual grounding and understanding in vehicle perception systems by reducing redundant visual computations.
Authors
Eunju Park, Uiwon Hwang
Abstract
Large Vision-Language Models (LVLMs) repeatedly process hundreds of visual tokens across decoder layers, incurring substantial computation and memory costs. Existing inference-time visual token pruning methods often irreversibly remove low-ranked tokens, although the set of visual tokens receiving high query-conditioned attention can change across decoder depth. Recoverable routing preserves skipped tokens for later reconsideration, but re-admits them with stale representations after bypassing intermediate transformations. We further find that replacing stale re-admitted states with their current-depth oracle states from a full-token reference trajectory substantially improves grounding performance. We therefore propose CatchUp, a training-free framework for depth-aware visual-token re-admission. CatchUp repeatedly re-evaluates query-conditioned visual relevance, processes only selected visual tokens through each decoder interval, and preserves deferred tokens for later re-admission. Upon re-admission, CatchUp approximates missed representation evolution using reference-to-current transition signals from semantically and spatially related visual tokens. Across three referring-expression grounding datasets and six multimodal benchmarks, CatchUp achieves the highest average relative retention in both benchmark groups at 64-, 128-, and 192-token budgets. At a 64-token budget with 88.9% fewer active visual tokens, CatchUp retains 92.17% of the full model's performance on average across six multimodal benchmarks while reducing end-to-end latency by 8.0%. Code is available at https://github.com/pej0918/CatchUp