Hybrid defense improves model safety or refusal after harmful tuning
Safer Content or Firmer Refusals? A Hybrid Perturbation Defense for Alignment under Harmful Fine-tuning
Cryptography and SecurityArtificial IntelligenceInformation RetrievalMachine Learning
Summary
When people fine-tune a safe AI model with new data, even a small bit of harmful information can make the model behave badly. The authors studied two ways to protect the model: one that changes the model’s internal representations and another that reduces bad changes to the model’s parameters. They combined these into a new method called VaccineBooster, which can either reduce harmful content or keep the model’s ability to say no clearly, but not both perfectly at the same time. Their work helps people choose the best protection depending on whether they want safer content or stronger refusal to harmful requests.
What this means in practice
- •For ai software developers: Defend large language models from harmful data during fine-tuning to maintain safer content in applications.
- •For enterprise ml ops teams: Preserve explicit refusal responses in aligned models exposed to untrusted fine-tuning by applying gradient-level defenses.
Authors
Muhammad Zeeshan Akram, Mufid Kamel Marican, Anvesh Reddy Yenugu, Ali Zain Kaimkhani, Minghong Fang
Abstract
Fine-tuning-as-a-service lets users adapt a safety-aligned language model to their own data, but it also creates a harmful fine-tuning attack surface: a small amount of harmful data mixed into an otherwise benign fine-tuning set can degrade the model's alignment. Two recent alignment-stage defenses address this problem at different levels of the model. Vaccine improves the robustness of hidden embeddings to the representation shifts induced by harmful fine-tuning, whereas Booster simulates harmful weight updates and attenuates their effect during alignment. We investigate whether these mechanisms are complementary and propose VaccineBooster, a single alignment procedure that combines embedding perturbation and weight-level gradient attenuation within each training step. On Llama-2-7B aligned with BeaverTails and then attacked through poisoned fine-tuning, VaccineBooster achieves the lowest OpenAI moderation score among the compared defenses, 0.315, while a Booster-Only variant retains the highest post-attack refusal rate, 50%. Together with ablations over the embedding-perturbation and gradient-attenuation strengths, these results indicate a trade-off: embedding perturbation primarily reduces flagged harmful content, whereas gradient attenuation primarily preserves explicit refusal behavior. Because our evaluation uses ten prompts and a single unseeded run per configuration, we report this trade-off as an observed pattern rather than a statistically resolved effect. These results provide practical guidance for prioritizing content safety or refusal retention when aligned models are exposed to untrusted fine-tuning.