Group-wise model merging improves multi-task AI performance
GRAM: Group-wise Rank-Aware Model Merging via Subspace Alignment
Artificial IntelligenceComputation and Language
Summary
Combining multiple AI models into one is useful but hard when each model learned different tasks from separate data. The authors found that merging models without considering the mathematical directions learned by each task can hurt performance. They propose GRAM, a method that groups similar tasks before merging, preserving important details. This approach works well when merging even just two models, improving results in language, vision, and medical tasks without needing the original data.
What this means in practice
- •For healthcare ai teams: Combine locally fine-tuned medical models into one while respecting data privacy and improving accuracy.
- •For machine learning engineers: Merge multiple task-specific models into a single model that handles all tasks better by grouping similar tasks.
Authors
Di Wu, Shuaidong Pan
Abstract
Merging different models into a single model is highly desirable for multi-task deployment, particularly in regulated domains like healthcare. In such settings, models are often locally fine-tuned due to strict data privacy policies, necessitating that aggregation operates exclusively on the fine-tuned weights. Existing methods apply a uniform merge across all tasks, ignoring the geometric relationships among their learned subspaces. When task updates occupy misaligned subspaces, projecting them onto a shared low-rank basis discards task-relevant directions, a failure mode named \emph{subspace interference}. Through this study, we show that the per-task projection error of a merged model is governed by the principal-angle alignment between each task and the shared subspace, so that grouping geometrically compatible tasks provably reduces this error within each group. Motivated by this observation, we propose \textbf{Group-wise Rank-Aware Model Merging (GRAM)}, which uses the principal-angle geometry of task vectors to decide which tasks should be merged together and then performs a standard SVD-based merge within each coalition; the pipeline requires only the fine-tuned weights at merge time. With only one additional model ($\nK{=}2$), GRAM significantly mitigates subspace interference and consistently outperforms prior merging methods across language understanding (GLUE with Flan-T5), 8-task CLIP vision, and medical instruction tuning (Qwen2-7B-Instruct with four English/Chinese medical LoRAs).