Mixture of experts model improves memory use by smart expert selection
Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
Machine LearningDistributed, Parallel, and Cluster Computing
Summary
Large AI models called Mixture-of-Experts (MoE) use many smaller models, or experts, but can't always fit them all in memory. The authors found that just picking the top experts independently misses how they work together. They created a method called DS-MoE that picks experts by balancing overlap and teamwork to keep important expert groups. Their approach finds better expert subsets that keep the model powerful but use less memory.
What this means in practice
- •For machine learning engineers: Improve memory efficiency of large sparse AI models by selecting expert combinations that reduce redundancy while maintaining complementary knowledge.
- •For ai system architects: Design MoE deployments that run efficiently on limited hardware by using optimized expert subsets that preserve key model functions.
Authors
Zheng Lin, Shaoke Fang, Yuxin Zhang, Jinfeng Xu, Zihan Fang, Zhe Chen, Wei Ni, Jun Luo, Symeon Chatzinotas
Abstract
While Mixture-of-Experts (MoE) models effectively scale model capacity through sparse activation, their deployment is often bottlenecked by prohibitive memory requirements. Extracting a compact subset of experts presents a promising solution. However, existing expert selection heuristics predominantly rely on Top-k ranking, which isolates the evaluation of individual experts and ignores the intricate inter-expert dependencies introduced by the MoE gating network. In this paper, we propose DS-MoE, a theoretically grounded framework that redefines expert selection via difference-of-submodular (DS) optimization. By analyzing the second-order Taylor expansion of the loss degradation, we reveal functional duality within expert combinations: redundancy (where experts encode overlapping representations) and synergy (where experts provide complementary error cancellation). To navigate this duality, we mathematically decouple redundancy reduction from synergy maximization by formulating the selection objective as a DS function. Furthermore, we devise a tailored majorization-minimization (MM) algorithm with provable monotonicity guarantees to efficiently identify the optimal expert subset. Extensive experiments demonstrate that DS-MoE effectively preserves indispensable expert combinations, achieving superior performance compared to the state-of-the-art baselines.