AI summaryⓘ
The authors explored whether very different large language models can be combined by simply averaging their parameters after adjusting their sizes, without extra training or complex alignment methods. They tried two ways: making the smaller model bigger or making the bigger model smaller, then mixing them in different proportions. They found that careful expansion and mixing can preserve useful abilities and sometimes improve performance by sharing strengths from both models. However, mixing too evenly often causes problems, with some skills improving while others get worse. This suggests simple averaging with size adaptation is a strong baseline for merging models, and more complex methods might not do much better.
large language modelsparameter averagingmodel mergingdimensional adaptationinterpolationQwen modelsbenchmark evaluationtransfer learningmodel fusionheterogeneous models
Authors
Jiahe Fan, Yinghao Hou, Si Chen, Aiyuan Zhang, Hong Xie, Defu Lian
Abstract
Can large language models with substantially different parameter spaces be merged by direct weighted averaging, without training or semantic alignment? Existing heterogeneous fusion methods typically introduce distillation, adapters, learned latent spaces, routing, or feature alignment, leaving open whether a simpler recipe can work for genuinely different billion-parameter checkpoints. We revisit this counterintuitive question through training-free dimensional adaptation followed by ratio-controlled interpolation. In union-style merging, we expand the smaller model into the larger parameter space; in intersection-style merging, we truncate the larger model into the smaller parameter space. Across Qwen-family model pairs and benchmarks covering mathematical reasoning, code generation, language understanding, commonsense reasoning, knowledge, and instruction following, deterministic expansion largely preserves the source model function, and small-ratio interpolation can improve over strong source checkpoints by transferring complementary capabilities. However, near-balanced interpolation often collapses, and task-level results reveal a seesaw effect in which gains on some capabilities coexist with regressions on others. These results show that simple parameter averaging, when paired with lightweight dimensional adaptation and carefully controlled ratios, is a surprisingly strong baseline for heterogeneous LLM merging, suggesting that the limits of direct weighted fusion may also bound what more complex heterogeneous merging methods can achieve at scale.