Multimodal large language models save compute with visual backtracking
ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Images in large language models are broken down into many pieces that computers use when answering questions, which takes a lot of computing power. The authors discovered that often pieces thought unimportant early on actually become helpful later for understanding the image better. They created a way called ReTrace that temporarily sets aside less important pieces instead of deleting them, then brings them back when needed. This method reduces computing work significantly while keeping the model’s accuracy high.
What this means in practice
- •For ai model developers: Build more efficient vision-language models by selectively pruning and restoring visual tokens during inference to save computation without losing accuracy.
- •For software engineers optimizing ml inference: Speed up multimodal AI systems for tasks involving image and text by cutting down calculation steps via reversible visual token caching.
- •For computer vision product teams: Develop faster and more resource-efficient AI products that handle complex image analysis combined with language processing using backtracking token techniques.$Commercial implications: This paper enables commercial AI products that offer faster image-text understanding with lower computational costs, appealing to deployment-focused technology companies.
Authors
Guangzong Si, Dong Wang, Yifan Yu, Miao Liu, Panwang Pan, Wentao Zhu
Abstract
Multimodal Large Language Models (MLLMs) encode images into dense visual token sequences, creating substantial attention and KV-cache overhead. Existing visual token compression methods reduce this cost by pruning low-saliency tokens in shallow layers, but this practice couples temporary computational deactivation with irreversible information deletion. We show that this coupling breaks an intrinsic inference pattern in MLLMs. Through layer-wise attention analysis, we identify \textbf{Visual Backtracking}: visual pivots remain spatially stable, yet in middle layers they actively retrieve context from tokens that appeared unimportant earlier. Thus, early low-saliency tokens can be inactive for computation while still being useful for later reasoning. Motivated by this finding, we propose \textbf{ReTrace}, a backtracking-aware compression framework that introduces a dormant token state. ReTrace caches non-pivot tokens outside expensive layer-wise computation, restores them during the backtracking window, and prunes only after their information has been absorbed. Experiments on Qwen2.5-VL and InternVL3 demonstrate nearly 90\% reductions in FLOPs and KV cache, a 2.83$\times$ prefilling speedup, and strong performance retention across complex vision-language benchmarks.