Modality aware pruning saves memory in multimodal large language models
Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models
Computer Vision and Pattern RecognitionComputation and Language
Summary
Large language models that handle both images and text use many expert components, which take up a lot of memory. The authors propose a way to shrink these models by carefully removing experts based on whether they are more important for image or text processing. Their method does this without needing extra training and keeps the model working well, especially on tasks involving both pictures and words. This makes it possible to run big models on smaller hardware.
What this means in practice
- •For machine learning engineers: Reduce memory use of large multimodal models to run on GPUs with limited memory by pruning vision experts selectively.
- •For cloud ai infrastructure teams: Lower infrastructure costs by compressing multimodal models to fit onto fewer or less powerful GPUs without major performance loss.
Authors
Donggeun Kim
Abstract
Mixture-of-Experts (MoE) enables multimodal large language models (MLLMs) to scale capacity efficiently, but expert parameters dominate memory even though only a few experts are activated per token: they account for over 90% of model size in modern MoE-MLLMs (e.g., 54 GB out of about 58 GB in BF16 for InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B). Expert pruning has emerged as an effective compression strategy, but existing methods aggregate importance uniformly across tokens, ignoring that the same expert may serve different roles for vision versus language processing. To address this, we propose MAEP (Modality-Aware Expert Pruning), a training-free framework that decomposes expert importance by modality in a single forward pass. MAEP introduces (1) Delta-H Output ($ΔH$), a redistribution-aware metric measuring the MoE output change when an expert is removed, and (2) low-attention vision tokens as a vision-side pruning signal that is dataset-stable: its expert-importance ranking remains stable across different datasets. Text-side importance serves as a preservation signal while vision-side importance serves as a pruning signal, and experts are ranked globally across layers. Experiments on three MoE-based MLLMs across multimodal and text benchmarks show that MAEP outperforms modality-agnostic baselines on the majority of multimodal benchmarks while remaining competitive on text-only tasks, reducing the BF16 expert weight footprint by about 27 GB (about 47% of total memory) so that the full weight footprint of an approximately 58 GB model fits within a single 40 GB GPU. Code is available at https://github.com/nota-github/MAEP.