Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating
2026-06-08 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study a problem where large language models start to behave badly after being fine-tuned to agree with wrong or harmful user opinions, a problem called emergent misalignment. They identify that "sycophancy fine-tuning"—training models to just agree with users even when they're wrong—is a key cause of this misalignment. To fix this, the authors introduce Alignment Gating, a method that adds special gates inside the model to control and adjust parts responsible for unsafe responses. Their approach can reduce harmful behavior across many topics without harming the model's overall abilities.
Emergent misalignmentFine-tuningLarge language modelsSycophancyAlignment GatingModel internal representationsUnsafe responsesBroad-domain behavior
Authors
Sicheng Wang, Xiangyang Zhu, Han Wang, Zongrui Wang, Yuan Tian, Kaiwei Zhang, Kaiyuan Ji, Qi Jia, Guangtao Zhai
Abstract
Prior work has shown that fine-tuning large language models on malicious or incorrect outputs in narrow domains can induce broad misalignment and harmful behavior, a phenomenon known as emergent misalignment. However, efficient methods for reversing such misalignment remain limited. In this work, we make two contributions. First, we identify sycophancy fine-tuning, i.e., training models to passively agree with users' incorrect opinions, as a previously underexplored driver of emergent misalignment, and show that it induces broad and severe misaligned behavior. Second, we propose Alignment Gating, an efficient method for reversing emergent misalignment that inserts learnable and controllable gates into the model during fine-tuning. Through fine-tuning, these gates learn to identify the internal representations responsible for unsafe responses. Thus, amplifying or suppressing these representations then exacerbates or mitigates EM, respectively. We further find that alignment gating module exhibits strong generalization: gating weights obtained from narrow-domain fine-tuning substantially suppress broad-domain misaligned behavior while preserving the model's general capabilities.