Fine-tuning safety depends on behavioral changes not just data
Safety Offset Learning: When Benign Fine-Tuning Misaligns LLMs
Machine LearningArtificial Intelligence
Summary
Fine-tuning large language models (LLMs) is usually judged safe or unsafe depending on the data used, but this paper finds that the way fine-tuning changes the model’s behavior is more important. The authors show that even when fine-tuning uses safe examples, it can cause the model to respond harmfully if it changes the model’s tendency to refuse certain requests. They create a dataset that triggers these behavioral changes and show it makes models less safe despite no harmful training data being involved. This reveals that safety depends not only on data but on which behavioral boundaries the fine-tuning shifts.
What this means in practice
- •For machine learning engineers: Test safety risks in aligned LLMs by examining behavioral boundary shifts during fine-tuning on benign data.
- •For ai safety teams: Monitor model refusal behavior changes to detect potential safety degradation even when fine-tuning uses non-harmful data.
Authors
Jesson Wang, Zhanhao Hu, Minghao Lin, Yuchen Wu, Aylin Caliskan, Mengyuan Li
Abstract
Safety risks from fine-tuning are commonly characterized by the harmfulness of the training data:harmful fine-tuning is considered dangerous, while benign fine-tuning is often assumed to preservealignment. We argue that this view misses a more fundamental factor-the behavioral transition inducedby supervision. Harmful fine-tuning and safety-degrading benign fine-tuning can both be understoodas learning the same refusal-to-compliance transition, but on different data distributions. We studywhether such a transition learned on benign inputs can generalize to harmful requests. We constructbenign examples that an aligned model initially tends to refuse and pair them with their originalharmless responses, creating a discrepancy between the model's initial refusal and the supervisedanswer. Fine-tuning on these examples increases harmful compliance, but only when two conditionshold together: the discrepancy is large enough, and it cannot be resolved through a narrow surface-levelshortcut. With the induced discrepancy matched, varying only whether it is localized changes harmfulcompliance by an order of magnitude; raising the discrepancy does not help while a shortcut remainsavailable, and removing the shortcut does not help while the discrepancy is small. Building on thisobservation, we introduce SAFEOFF, a dataset of benign instructions rewritten to raise the alignedmodel's refusal likelihood and paired with their original answers. Across three aligned LLMs and threesafety benchmarks, SAFEOFF substantially weakens safety alignment despite containing no explicitharmful supervision, suggesting that fine-tuning safety depends not only on what data a model sees, butalso on which behavioral boundary the supervision moves.