Model coefficient transfer speeds up and cuts memory use in merging
$α$Transfer: Coefficient Transfer for Efficient Model Merging
Machine LearningComputation and LanguageComputer Vision and Pattern Recognition
Summary
Combining different versions of AI models is tricky because finding the best way to mix their settings takes a lot of time and memory. The authors found that you can figure out the best mixing ratios on a smaller model, then reuse those ratios on bigger models in the same family. This approach, called αTransfer, is much faster and uses less memory without losing performance. It works across many types of models and tasks.
What this means in practice
- •For machine learning engineers: Accelerate combining multiple fine-tuned checkpoints into one model with less memory and time use.
- •For ai service providers: Cut computational costs when deploying merged models at scale by reusing optimal mixing coefficients found from small proxy models.$Commercial implications: Provides efficient model merging techniques that lower infrastructure costs for AI services handling large models.
Authors
Shih-Cheng Huang, Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Hung-yi Lee, Shao-Hua Sun
Abstract
Model merging offers a promising solution for combining multiple fine-tuned checkpoints into a single model through parameter arithmetic. However, finding optimal merging coefficients requires an extensive search that becomes prohibitively expensive as models scale in both size and number, due to high memory requirements and combinatorial growth in the search space. We show that, within the same model family, models exhibit highly congruent performance distributions over merging coefficients across different model sizes. This distributional similarity enables a practical paradigm we call \textit{$α$Transfer}: searching for optimal coefficients on a small proxy model, then directly transfer them to larger target models. We verify $α$Transfer across multiple merging methods, model families, and tasks. Experimental results demonstrate a 6$\times$ speedup and 70\% memory reduction on vision transformers, and a 20$\times$ speedup and 85\% memory reduction on large language models, while maintaining comparable performance. Our findings establish $α$Transfer as an efficient and generalizable approach to scaling model merging.