Routing changes in merged MoE models do not always mean failure
Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs
Artificial IntelligenceMachine Learning
Summary
When combining specialized language models that use mixtures of experts (MoE), changes in which expert handles parts of the input — called routing drift — often happen. The authors studied whether these routing changes actually cause model failures or just reflect normal shifts in input. They found that routing drift alone does not clearly predict drops in performance or the need for fixing. Instead, whether a routing change is harmful depends on how it affects the overall task, not just the routing pattern. They also provide tools and a repair method to test and intervene in routing decisions.
What this means in practice
- •For machine learning engineers: Test merged Mixture-of-Experts models for routing-related failures using task-level interventions, rather than relying on routing pattern changes alone.
- •For ai system integrators: Improve deployment safety of combined specialized language models by applying token-level routing analysis and selective repairs to maintain task performance.
Authors
Yuanyi Wang, Yanggan Gu, Su Lu, Guanghao Zhu, Pengkai Wang, Yifan Yang, Congkai Xie, Zhaoyi Yan, Jianmin Wu, Hongxia Yang
Abstract
Model merging efficiently combines specialized large language models (LLMs) without joint retraining, but can substantially alter expert routing in Mixture-of-Experts (MoE) models. Such \emph{routing drift} is often interpreted as routing failure, raising a fundamental question that remains unclear: \emph{does routing drift after MoE merging actually indicate routing failure, and what evidence should justify repair?} We investigate these questions across DeepSeekMoE, OLMoE, and Qwen3-MoE proposing a routing analysis toolkit for controlled counterfactual interventions and token-level analysis. By crossing source and merged router inputs and parameters, we attribute most expert reassignments to input shifts rather than parameter changes at the same layer. However, source-relative routing differences poorly predict next-token likelihood gains from source-route restoration, and different expert selections can produce directionally similar mixture outputs. We therefore operationalize routing failure as \textit{task loss recoverable under a specified routing intervention, with non-routing parameters fixed.} These tests detect recoverable loss under deliberate router corruption, whereas source-route restoration does not establish reliable task benefits in the evaluated merged models. Motivated by these, we propose \emph{Selective Router Repair (SRR)} as a case study, and find that source-specialist token-likelihood advantages do not reliably identify beneficial local corrections. Together, these findings show that \textbf{routing drift alone is insufficient evidence of routing failure}: source-informed corrections must be judged by their task-level intervention effects. The analysis toolkit and SRR code are released.