Maskerade improves model efficiency by routing token-specific subnetworks
MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling
Machine Learning
Summary
Large language or vision models are often big and slow to run because they use the same parts for every task. The authors found a way to speed things up by splitting a big model into smaller parts called experts, each focused on a different subset of neurons inside a frozen pretrained network. Their method, MASKerade, picks which expert to use for each token (a piece of input text or image) without changing the original weights. This approach saves computation and works well on tests involving both images and language.
What this means in practice
- •For machine learning engineers: Create more efficient language or vision models by building experts as sparse subnetworks over frozen pretrained networks, reducing computation without extra parameters.
- •For ai infrastructure teams: Implement systems that dynamically route input tokens to specialized sparse subnetworks to reduce inference costs on large pretrained models in production.
Authors
Mingyuan Zhang, Yue Bai, Zhongruo Wang, Yupin Huang, Yiyang Huang, Hailing Wang, Huimin Zeng, Yun Fu
Abstract
Sparsely activated Mixture-of-Experts (MoE) models increase model capacity without a proportional increase in per-token computation. Dense-to-MoE upcycling reuses pretrained dense models to construct such systems, commonly by copying feed-forward networks (FFNs) into independently trained experts. We introduce MASKerade, a dense-to-MoE training method that instead learns experts as sparse subnetworks of a frozen pretrained FFN. Each expert is defined by a learned binary mask, and a token-level router selects which masked FFNs to execute and combine. The router and mask scores are optimized jointly, while the underlying FFN weight values remain unchanged. This formulation supports neuron-structured, semi-structured, and unstructured experts within the same routing architecture. Our main configuration uses four 2:4 experts with top-2 routing, where two half-dense expert passes have the nominal FFN arithmetic of one dense pass, without requiring independent expert weight matrices. On five vision-language benchmarks with Qwen and Gemma backbones, this configuration achieves the highest performance among the compared baselines. Comparisons across mask granularities, routing interventions, and compute-matched controls distinguish the effects of learned connectivity from expert activation count. These results establish mask learning over frozen weights as a practical alternative for constructing token-routed MoE experts.