Fisher merging optimizes multi-teacher model consolidation efficiently
A Fisher-Geometric View of Multi-Teacher On-Policy Distillation
Machine LearningComputation and Language
Summary
Combining specialized AI models into one general model is challenging due to conflicts and high computation needs. The authors found that two popular approaches to this problem are mathematically linked, showing that a classic fast merging method is actually the best second-order solution. They propose a new two-step method, RGBM, that uses this insight to efficiently combine models with less training. Their technique matches or beats existing methods on real model sets without extra training time.
What this means in practice
- •For machine learning engineers: Combine multiple specialized AI models into a unified model more efficiently with less computational overhead.
- •For ai platform operators: Optimize deployment of specialized model ensembles by consolidating them with fewer joint training steps and maintaining performance.
Authors
Liu Kaiyuan, Zhuang Ziyuan, Ma Haoxuan, Gu Yannian, Song Mingfei, Weng Rongxiang, Ye Jieping
Abstract
Consolidating specialized foundation models into a unified generalist is a central goal of post-training. Two approaches are in common use: model merging, which is fast but treated as a heuristic, and multi-teacher on-policy distillation, which matches distributions but demands heavy rollout compute and suffers from cross-task gradient conflict. We show that the two are one objective seen at different orders. Within a shared basin of homologous specialists, the multi-teacher on-policy distillation objective is, to second order, exactly the Fisher-merging objective. Its minimizer is therefore the Fisher merge, a classic model-merging method given by a closed-form combination of the specialists, so merging is optimal, to second order, as a closed-form stage of distillation. Building on this identification, we propose Residual-Governed Branch-and-Merge(RGBM), a two-phase consolidation procedure. Concretely, a full-parameter Fisher merge is intractable on foundation models, so Phase~1 restricts the merge to the task-vector subspace. Extending tangent-space analyses of merging, we show that the same coefficients that mix task vectors also mix implicit rewards, which lets a residual program on unlabelled prompts solve for them. The program minimizes the worst per-domain residual and reports its energy-normalized values as the domain residual spectrum. Phase~2 lets this spectrum govern what may be added: RGBM passivates domains with low normalized residuals, trains isolated branches only for domains above the spectral gap, and writes back the admitted updates with calibrated shrinkage. On two open-source consolidation suites, the Phase~1 merge alone matches or outperforms fully trained multi-teacher distillation, and RGBM improves on it further with zero extra joint-training steps, whereas every un-gated write-back, including continued joint distillation from the merge, falls below it.