Model merging preserves some but not all emergent AI abilities
On Emergent Capabilities and Model Merging
Machine LearningArtificial Intelligence
Summary
This paper studies what happens when two AI models are combined by mixing their internal settings. The researchers found that if both original models share a surprising behavior, the combined model keeps it. But if only one model has the behavior, merging usually weakens it, and merging two models with different special abilities won't create a better combined ability. Basically, surprising behaviors in AI models don’t mix as easily as the tasks they were trained for.
What this means in practice
- •For machine learning engineers: Combine misaligned AI model checkpoints while retaining their shared unintended behaviors to understand risks in model merging operations.
- •For ai safety teams: Evaluate how merging different AI components affects hidden risky behaviors for safer integration of fine-tuned models in production.
Authors
Luca Zhou, Emanuele Rodolà
Abstract
Fine-tuned checkpoints and adapters now fill public repositories, and the most common operation applied to these artifacts is model merging: arithmetic on their weights that assembles capabilities cheaply. We ask what this operation does to emergent capabilities: behaviors an artifact carries that were never an explicit training target. Studying two independent testbeds (activation oracles and emergent-misaligned models) across three model families, we find that the answer is threefold. First, merging preserves an emergent capability that both parents carry: merging two misaligned checkpoints retains most of their broad misalignment across the whole mixing range. Second, merging cannot create an emergent capability that is superadditive in its parents: no weighted merge of two single-task oracles reaches the jointly-trained oracle's auditing ability. Third, when only one parent carries the capability, merging dilutes it faster than the trained capability that accompanies it: the gap is significant in most settings. In short, emergent behaviors of an artifact do not compose the way its trained capability does.