Mathematical guarantee for training mixtures of linear expert models
Provably Learning Mixtures of Linear Experts
Machine Learning
Summary
Mixture-of-Experts (MoE) models break complex tasks into parts handled by different expert models, but training them is tricky and can get stuck. The authors analyze a basic form of MoE where the input groups follow Gaussian patterns, proving for the first time that a common training method can find the best solution from random starts under certain conditions. Their math breaks the problem into simpler parts for easier analysis, and experiments show their approach matches predictions and works well even without some technical starting steps. They also find ways to simplify the model by focusing on the truly used experts without losing accuracy.
What this means in practice
- •For machine learning engineers: Provide confidence in using gradient EM to jointly train linear MoE models with gated inputs, improving model reliability under specific cluster separation conditions.
- •For software developers in ai tools: Enable efficient pruning of overparameterized MoE models by identifying and preserving key experts, reducing model size without sacrificing performance.
Tested on simulated data.
Authors
Adrian Candocia, Aamod Varma, Mark Davenport
Abstract
Mixture-of-Experts (MoE) models are widely deployed in modern machine learning. However, few convergence guarantees exist for gradient-based training of MoEs: the objective is nonconvex, and even simple mixtures trap first-order methods in spurious local optima. We analyze a canonical gated mixture featuring linear experts and a gate induced by Gaussian clusters in the input space, and give, to our knowledge, the first global convergence guarantee for jointly training the gate and the experts of such a model by gradient EM on its likelihood. Given a logarithmic overparameterization relative to the true number of experts $m$, gradient EM converges from randomly initialized gates and least-squares-initialized experts to any prescribed accuracy in polynomially many iterations, under a separation condition on the gate centers. Our analysis establishes a gradient-dominance inequality for the coupled dynamics by decoupling the loss into per-cluster subproblems. Empirical results validate the predicted $m\log m$ phase transition and suggest that the least-squares warm start required by our analysis is not needed in practice: randomly initialized experts perform nearly as well. We also find that leaving the mixing weights unconstrained concentrates routing onto the true number of experts, whereas load-balancing penalties spread it, so that the fitted model can be pruned to a fraction of its width without loss of accuracy.