Efficient multimodal model inference speeds visual language tasks
DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
Computer Vision and Pattern Recognition
Summary
Multimodal AI models that work with both images and text are powerful but slow because they do too much work on visual parts repeatedly. The authors found two main ways to speed this up: by stopping visual processing early when possible, and by reorganizing the experts that process text tokens to keep only the most important parts. Their new approach, called DecoMoE, speeds up the model without much loss in performance by cutting down unnecessary computations both visually and in the expert process. This results in a faster system that still performs almost as well as before.
What this means in practice
- •For ai engineers: Reduce inference time and computational cost in large multimodal AI systems to enable faster real-time image and text understanding.
- •For cloud service operators: Lower the resource demands of serving multimodal AI models, cutting operational costs for cloud-based AI inference services.
Authors
Xudong Tan, Peng Ye, Ming Xie, Chenyu Huang, Yaoxin Yang, Jiayuan Fan, Tao Chen
Abstract
Multimodal mixture-of-experts (MoE) models combine sparse expert activation with visual-language capabilities, yet their inference remains costly because long visual-token sequences repeatedly incur attention, routing, dispatch, and expert-MLP computation. Existing methods typically compress either the token or expert dimension, leaving redundancy along the other. Our analysis reveals two complementary regularities: the depth required for visual propagation varies across inputs, while text-token routing exhibits concentrated and recurrent expert-importance patterns. Based on these observations, we propose DecoMoE, a two-dimensional structured compression framework that decouples visual propagation from expert computation. The Sample-Adaptive Visual Boundary (SAVB) predicts an input-dependent visual-exit layer at which the visual-token block is removed. The Routing-Calibrated Expert Prefix (RCEP) reorders experts offline using text-token routed mass and, from this predicted exit layer onward, retains at each MoE layer the shortest contiguous prefix covering a target routed-mass fraction. We evaluate DecoMoE on Qwen3-VL-MoE and InternVL3.5-30B-A3B across six benchmarks. On Qwen3-VL-MoE, DecoMoE retains 97.91% of dense-baseline performance while reducing computation from 27.06 to 16.73 TFLOPs and latency from 0.44 to 0.26 seconds, yielding a 1.69x speedup. Code will be available at https://github.com/ShawnTan86/DecoMoE.