Energy-based method improves merging of multi-task AI models
Not All Task Vectors Need Equal Rank: Energy-Proportional Allocation for Model Merging
Artificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
Summary
When combining several AI models trained on different tasks, it's tricky to merge them into one model that does all tasks well. The authors found that treating all tasks equally during merging isn't the best, because some tasks are more complex than others. They created a method called SERA that gives each task a share of the model's capacity based on how complex its task changes are. This approach makes the merged model perform better on multiple tasks without needing more resources.
What this means in practice
- •For machine learning engineers: Build multi-task models from fine-tuned models with better performance using adaptive rank allocation in spectral merging.
- •For machine vision developers: Improve merged vision models that handle multiple tasks without increasing model size by allocating spectral capacity according to task complexity.
Authors
Hyunjoong Cho, Jinhyeok Jang
Abstract
Model merging aims to combine multiple fine-tuned models derived from a common pretrained model into a single multi-task model without additional joint training. Recent spectral merging methods improve over simple weight averaging by exploiting low-rank structures of task-specific updates, but they commonly assign the same rank capacity to every task. This uniform allocation ignores that task vectors can have heterogeneous spectral complexity, causing the shared merging space to be used suboptimally. In this paper, we propose Spectral Energy-proportional Rank Allocation (SERA), a simple task-adaptive strategy that allocates ranks according to the singular-value energy structure of each task vector. By assigning richer spectral capacity to complex or isolated tasks and fewer directions to compact tasks, SERA extends SVD-based model merging from uniform-capacity merging to task-dependent capacity allocation. Experiments under standard vision model merging protocols show that SERA improves multi-task merging performance while preserving the same total rank budget as existing spectral merging methods. Further analysis demonstrates that task-level spectral concentration is closely related to the per-task effect of adaptive rank allocation, providing insight into when and why SERA is effective.