Adaptive skipping reduces computation in multimodal large language models
AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Multimodal large language models process many small pieces of visual information, which takes a lot of computing power. The authors discovered that not all parts of the model need to handle every visual piece equally, and this varies between inputs and model components. They created AdaVSkip, which uses small decision-makers in each part of the model to skip unnecessary visual computations for each input. Their method keeps the model’s accuracy almost the same while cutting the computations significantly, making it faster and more efficient.
What this means in practice
- •For mobile app developers: Speed up multimodal AI models on smartphones by reducing unnecessary visual processing per input.
- •For cloud ai service operators: Cut the computational costs of running large multimodal language models in data centers while maintaining accuracy.
Authors
Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang
Abstract
Multimodal large language models (MLLMs) require substantial computation to process numerous visual tokens across all transformer layers. Most methods for efficient MLLM inference exploit horizontal redundancy by compressing visual tokens. Beyond token reduction, recent studies exploit vertical redundancy through early exit or fixed-layer skipping. However, we find that the extent and distribution of this redundancy vary across inputs and differ between self-attention and MLP modules. Motivated by these observations, we propose AdaVSkip, which equips each layer with two lightweight routers that independently determine whether visual tokens pass through by or skip the self-attention and MLP modules. These decisions collectively define an input-specific visual-computation path, but their discrete and non-differentiable nature makes learning effective paths challenging. To address this challenge, we develop a progressive two-stage training framework that updates only the routers while keeping the backbone frozen. Stage I establishes an initial routing policy through supervised training with input-specific targets derived from module-wise necessity scores. To further align the routing policy with task performance, Stage II uses reinforcement learning to optimize routing decisions with direct feedback from generated answers. It combines an answer correctness reward with a skip-consistency reward that discourages excessive retention of visual-token computation. Across three MLLM backbones, AdaVSkip maintains strong task performance with substantially less computation. On LLaVA-NeXT-7B, AdaVSkip reduces FLOPs by 53.2\% while preserving the original model's average performance. Combining it with visual token compression increases this reduction to 91.2\%, while retaining 97.2\% of the original performance on average.