Backdoor attack triggers only after merging large language models
Towards Merging-conditional Backdoor Unalignment against Large Language Models
Cryptography and SecurityArtificial IntelligenceComputation and Language
Summary
Large language models can be made better by merging different versions, but this study shows a hidden risk: a backdoor that stays invisible in one model but activates once models are merged. The authors show this happens because merging shifts model settings past a safety boundary. They then create a way to implant such a hidden backdoor using a method called SMOKE. Experiments prove the attack works across various model types and merging methods and is hard to detect with current tools.
What this means in practice
- •For ai security teams: Detect and defend against hidden backdoors that activate only after merging language models to improve AI safety.
- •For model integration engineers: Evaluate and verify merged models to avoid unexpected behavior caused by dormant backdoors triggered by merging.
Authors
Yukun Chen, Yidou Chen, Yiming Li, Xinyu Zhang, Baosong Yang, Hongwei Yao, Zhan Qin
Abstract
Model merging has become widely adopted for enhancing the task-specific performance of Large Language Models (LLMs) without additional training. While the specialized model from third-party platforms may undergo safety audits (e.g., backdoor detection), we reveal a novel and critical threat: One-sided Merging-Conditional Backdoor (OMCB). OMCB remains dormant and undetectable within one standalone model but activates upon merging, triggering safety unalignment. We first theoretically analyze the existence and geometric location of OMCB, revealing that the parameter shift crossing safety decision boundaries is the main cause of backdoor activation. Building upon this insight, we propose SMOKE, a meta-learning-based attack framework. SMOKE implants OMCB via a bi-level optimization strategy: the inner loop synthesizes surrogate meta-vectors through random low-rank initialization and finite-step maximization, while the outer loop optimizes the target model by simulating the merging process. Furthermore, preliminary analysis shows that the injected OMCB has bounded generalization loss under unknown merging configurations. Extensive experiments across diverse model architectures and merging techniques validate the effectiveness of SMOKE and its resistance to existing backdoor detections, highlighting a significant yet previously overlooked vulnerability in the model merging paradigm.