DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
2026-05-11 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors present DECO, a new type of Mixture-of-Experts (MoE) model that uses fewer active parts during computation to save memory and speed up processing without losing accuracy. They designed a special way to route data through the model using ReLU functions and scaling to balance different experts' contributions. They also introduced a new activation function called NormSiLU to make the model more stable and sparse. In tests, DECO performed as well as traditional dense Transformers but was faster and more efficient, activating only 20% of its experts.
Mixture-of-ExpertsTransformerReLUActivation functionNormSiLUSparse modelsRoutingModel efficiencyInference speedParameter budget
Authors
Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Zhiyuan Liu
Abstract
While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens. DECO utilizes the differentiable and flexible ReLU-based routing enhanced by learnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduce NormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing a more stable trend of routed-expert activation ratio and a higher intrinsic sparsity level. We also identify an empirical advantage in using non-gated MLP experts with ReLU-based routing, indicating the possibility of MoE architecture simplification. Experiments demonstrate that DECO, activating only 20% of experts, matches dense performance and outperforms established MoE baselines. Our specialized acceleration kernel delivers a 3.00$\times$ speedup on real hardware compared with dense inference. Codes and checkpoints will be released.