Signature-Guided Capacity Occupancy for Dense Expert Merging
2026-08-10 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors study how to combine special-purpose language models into one model efficiently. They identify three key challenges in this process: deciding where to add extra capacity in the model, figuring out which domain should get that capacity, and doing this without expensive trial-and-error methods. They propose SigMerge, a new approach that uses conflict signatures to assign capacity per layer and domain, then sequentially merges model parts within the assigned budgets. Experiments show that their method consistently improves performance across many merging scenarios compared to existing techniques.
language modelsmodel mergingdense expert mergingcapacity assignmentcross-expert conflictdomain specializationweight spacelayer capacitymodel checkpoints
Authors
Lingching Tung, Chi-Jui Kim, Beicheng Xu, Yuchen Wang, Bin Cui
Abstract
Dense expert merging combines domain-specialized language models into one single checkpoint, typically by admitting task-vector support in weight space. However, this admission is governed by three decisions that existing methods answer only partially: where to open layer capacity from cross-expert conflict, who should occupy that capacity based on domain demand, and how to admit the resulting support without relying on costly recipe search. To tackle these issues, we propose SigMerge (Signature-Guided Capacity Occupancy), a structured capacity assignment framework for dense expert merging. Starting from a dense base merge, conflict signatures set each layer's capacity from cross-expert conflict, positive base-merge deficits set each domain's share of that capacity, and a sequential occupancy rule admits each expert delta up to the resulting layer-domain budget. Across 21 paired settings spanning seven dense base merges and three model pools, SigMerge improves every one (by 15.0% on average) and achieves the best average rank (1.67) among six merging methods, outperforming three categories of merging baselines.