Heterogeneous model merging improves specialist knowledge transfer
Exploring Heterogeneous Model Merging Approach for Complex Knowledge Transfer
Computation and LanguageMachine Learning
Summary
Specialized AI models are good at particular tasks, but it's usually hard to copy their skills directly into bigger general models without extra training. The authors tested two methods that combine parts of a specialist model into a general one without retraining or aligning their data. They found both methods helped the general model perform better on tasks like rewards and reranking. This means it’s possible to share skills between different AI models more simply than before.
What this means in practice
- •For machine learning engineers: Integrate specialized task capabilities into general AI models without retraining by merging parameters directly.
- •For ai system integrators: Improve general language model performance on specific functions such as reranking or reward modeling via simple parameter injections.
Authors
Jiahe Fan, Si Chen, Yinghao Hou, Wenbo Xia, Ke Xu, Hong Xie, Enhong Chen
Abstract
Specialized models encode task-oriented behavior, but transferring that behavior to a general language model usually requires training, distillation, or representation alignment. We study whether such ability can instead be transferred directly at the parameter level. We apply two existing training-free heterogeneous merging methods, previously shown to transfer knowledge between general language models, to specialist-to-general transfer, projecting a specialist donor into the recipient's shape and interpolating backbone parameters without gradient updates or semantic alignment. Intersection-Merge (IM) injects a prefix-aligned donor slice matching the recipient shape, while Activate-Prune-Merge (APM) uses forward-pass activation statistics to select which donor dimensions to retain before injection. Across embedding, reranking, reward modeling, and MoE code-specialist transfer, both methods improve the general recipient, showing that simple heterogeneous merging can move capabilities across diverse specialist roles.