Rethinking Expert Training for Model Merging with Prompt Learning
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors studied how to better combine multiple specialized AI models into one that can handle several tasks. They found that using prompt-based adaptation, where task-specific instructions are learned without changing the main model, works well and avoids problems when merging. They then introduced a new two-step training method called Dual-Tuned Experts (DTEs) that first trains prompts and then fine-tunes the model backbone, leading to better compatibility when merging. Their experiments showed that DTEs improve the performance of combined models across different setups.
model mergingprompt-based adaptationfine-tuningDual-Tuned ExpertsCLIPLoRAvision encodermulti-task learningtask interferenceparameter updates
Authors
Christos Georgakilas, Aniello Panariello, Samir El Karrat Moreno, Simone Calderara, Dimosthenis Karatzas, Joost van de Weijer
Abstract
Model merging aims to combine multiple domain-specialized experts trained from a shared foundation model into a single multi-task model. Existing approaches largely focus on improving the merging procedure itself and typically assume experts obtained through full-parameter fine-tuning. In this work, we revisit expert training for model merging. We first show that prompt-based adaptation provides a strong baseline: independently learned prompts can be exploited across tasks while keeping the backbone fixed, avoiding the interference introduced by weight merging. Building on this observation, we introduce Dual-Tuned Experts (DTEs), a two-stage training strategy that first learns prompts and then fine-tunes the vision encoder. This reduces the magnitude of task-specific parameter updates and produces experts with higher merge compatibility. Experiments across multiple CLIP architectures, full fine-tuning, and LoRA experts show that DTEs consistently improve merged performance of standard merging approaches and remain effective even when combining heterogeneous sets of experts.