Experts specialise in combined fine features for better model routing
RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Large AI models use parts called experts that handle different tasks. It was believed each expert focuses on one big topic, but the authors suggest each expert actually handles many small mixed features instead. They created a method called RouterInterp that better explains which details guide the model to choose an expert. This helps people understand how these complex AI models make decisions more clearly.
What this means in practice
- •For machine learning engineers: Debug and improve mixture of experts model routing by understanding which detailed features activate each expert.
- •For ai infrastructure teams: Monitor and optimize multi-expert AI systems by interpreting routing decisions with higher accuracy.
Authors
Ilya Lasy, Nora Yinuo Cai, Kola Ayonrinde
Abstract
Sparse Mixture of Experts (MoE) models scale more efficiently than dense models by routing tokens to modular expert networks that are only active for processing a fraction of tokens. A leading hypothesis for the performance of MoE models is that each expert specialises in a single, coherent domain. However, interpretability efforts that assume this hypothesis have generally been unsuccessful. We propose and present evidence for an alternative account that we call the Superposed Specialisation Hypothesis (SSH): experts specialise in a disjoint union of fine-grained features rather than one broad domain. Leveraging the SSH, we introduce RouterInterp, a method for interpreting expert routing that identifies Sparse Autoencoder features most predictive of routing decisions and produces unified natural language explanations. On gpt-oss-20b, RouterInterp explains expert routing with ${\sim}65\%$ higher detection accuracy than prior token statistics based methods. This work provides a scalable method for generating more accurate explanations of expert routing and increases our understanding of a previously uninterpretable component of foundation models.