Summary
The paper studies how to safely remove parts of visual input in language models that process both images and text without needing additional training. The authors found that deciding which image tokens to remove depends not just on how important each token seems but also on where in the processing the removal happens and which other tokens are removed at the same time. They introduce a new two-step method called CoRePrune that adapts token removal as the model processes the image and text, preserving model accuracy even when many tokens are removed. This approach speeds up the model while keeping its performance on tasks involving images and videos.
What this means in practice
- •For machine learning engineers: Reduce computational cost and latency in multimodal LLMs by efficiently pruning visual tokens without retraining.
- •For video processing teams: Maintain accuracy in video understanding tasks while aggressively reducing visual token inputs for faster analysis.
Authors
Shengli He, Yongchao Liang, Roumeng He, Junjie Zeng, Jiyuan He, Xin Fang, Can Wu, Li Zheng
Abstract
Training-free visual-token pruning often uses token importance, redundancy, or related selection criteria as proxies for safe removal. We show that these signals alone do not fully characterize removability, which is conditioned on both representation depth and the surrounding deletion set. Controlled interventions demonstrate that removing the same tokens at different depths produces substantially different downstream perturbations, while changing only the deletion context at a fixed depth alters candidate marginals and pruning-boundary decisions. These findings show that token importance alone cannot determine when a token is safely removable or how its removability changes under joint deletion. Motivated by this perspective, we propose CoRePrune, a training-free two-stage framework. Progressive Perturbation-Aware Visual Pruning refreshes deletion effects as visual representations evolve, while Set-Conditioned Refinement reevaluates candidate rescue benefits under the current deletion set after visual--text interaction. Across five multimodal large language model backbones covering standard images, high-resolution inputs, and video, CoRePrune preserves performance under aggressive token budgets. On Qwen3.5, with a final budget of 128 visual tokens, it retains 90.3% of dense-model performance while reducing aggregate prefill time by 51.0%.