Summary
This paper looks at how choosing only a few prediction models (experts) to use for each input affects how well those models can be separated and understood. The authors show that when experts group well, the difficulty depends on the size of the biggest group, but if there are no clear groups, it depends on the total number of experts. When routing is sparse, meaning only a few experts are used, the complexity depends on how many are active and the probability of ignored areas. Notably, using only the top expert chosen (Top-1 routing) lets models be recovered without worrying about how separated they are. They combined mathematical tools to reach these conclusions.
What this means in practice
- •For machine learning engineers: Optimize sparse expert model designs by selecting fewer active experts to improve recovery and reduce routing complexity in predictive systems.
- •For data infrastructure teams: Inform routing strategies that balance computational cost and expert separation by understanding how sparse routing impacts expert recovery.
A theory result. No direct application yet.
Authors
Huy Nguyen, Dung Le, Alessandro Rinaldo, Nhat Ho
Abstract
Sparse mixtures of experts combine multiple prediction models while evaluating only a small subset for each input. Although this architecture reduces computation, how expert separation and routing sparsity jointly govern expert recovery remains an open problem. In this paper, we address this question for location Gaussian mixtures of experts with a fixed number of experts by establishing how expert separation and spatial arrangement determine recovery rates. For dense models, our results reveal that when experts form well-separated clusters, the separation geometry is governed by the size of the largest cluster; without a prescribed cluster structure, this geometry is instead driven by the total number of experts. For sparse models, the separation geometry on retained routing regions is determined by the number of active experts, while the contribution of omitted regions is controlled by their total probability. Consequently, activating fewer experts reduces the separation complexity in these bounds. Furthermore, Top-1 routing yields a particularly striking result: experts can be recovered at a rate independent of the separation quantity. Our technical approach combines polynomial interpolation and Gaussian reproducing-kernel Hilbert space (RKHS) test functions with quantitative restriction inequalities that extend pointwise comparisons to expert functions.