Joint weight perturbations improve resistance to harmful model fine-tuning
When Safety Fails Collectively: Robust Alignment Against Harmful Fine-Tuning via Joint Weight Perturbations
Artificial Intelligence
Summary
Large language models can become unsafe if someone fine-tunes them with harmful changes later on. The authors found that when multiple small harmful changes happen together, their combined effect is much worse than the sum of each one alone. To fix this, they trained the model during its safety alignment phase to handle different combinations of harmful changes, making it less sensitive to such harmful fine-tuning later. Their method helps keep models safer without hurting how well they perform on tasks.
What this means in practice
- •For machine learning engineers: Enhance model safety during training by exposing models to combined harmful perturbations reducing risks in downstream fine-tuning.
- •For ai product teams: Improve reliability of AI applications by deploying language models better protected against safety failures caused by harmful updates.
Authors
Zhiyuan Wang, Yuezhi Zhou, Zhu Zhang, Zhixing Tan
Abstract
Harmful fine-tuning can rapidly erode the safety alignment of large language models, creating substantial risks when aligned models are adapted to downstream tasks. Existing alignment-stage defenses use harmful examples and simulated harmful perturbations during safety alignment to prepare models for later harmful fine-tuning, yet the collective effects of harmful weight perturbations remain underexplored. We identify a collective failure pattern: disjoint parts of a harmful perturbation have limited effects in isolation, but their joint application increases the model's harmfulness beyond the sum of their individual effects. Guided by this observation, we propose Joint Weight Perturbations (JWP), an alignment-stage defense that masks different parts of a harmful perturbation estimated from the current model and trains the model to respond safely in the resulting states. This exposes the model to different combinations of harmful perturbations and reduces its sensitivity to their accumulation during subsequent harmful fine-tuning. Experiments across downstream tasks and model families show that JWP offers stronger resistance to harmful fine-tuning than prior defenses while largely preserving task performance.