Model merging improves by adapting to task differences and depth
Beyond Uniform Subspaces: Spectrum-Aware and Depth-Adaptive Fusion for Multi-Task Model Merging
Computer Vision and Pattern Recognition
Summary
Combining models trained for different tasks usually treats all tasks the same, which can miss important differences. The authors found that tasks vary in how complex their updates are and how sensitive they are to changes across different layers of the network. They created a new merging method that adjusts to these differences, improving how multiple task models can be combined without extra training. Their approach works well on two popular types of model updates and shows consistent gains in experiments.
What this means in practice
- •For machine learning engineers: Combine task-specific neural networks more effectively without retraining by adapting merging methods to individual task characteristics and network depth.
- •For ai platform developers: Enhance AI model deployment pipelines by enabling better fusion of models trained on different tasks without requiring additional data or retraining steps.
Authors
Ruxi Gu, Zilei Wang, Wei Wang
Abstract
Model merging aims to consolidate multiple task-specific models without access to extra training process. However, existing subspace-based methods largely rely on a uniform treatment of task updates, overlooking their intrinsic spectral and depth-wise heterogeneity. We identify two key deviations from this assumption: different tasks require different subspace capacity and exhibit different tolerance to spectral transformation, while subspace projection introduces depth-dependent distortion. Based on these observations, we propose SADA-Merging, a spectrum-aware and depth-adaptive framework for data-free model merging. SADA-Merging allocates task-specific subspace capacity according to spectral complexity, adapts spectral preservation according to task-wise plasticity, and applies depth-dependent anchoring to compensate for projection-induced distortion. This enables the fusion process to adapt to both the intrinsic geometry of each task and its sensitivity across network depth. SADA-Merging operates directly on task updates and is applicable to both full fine-tuning and LoRA settings. Extensive experiments demonstrate consistent improvements over existing data-free merging methods across different task scales and adaptation settings.