Fine-tuning language models safely with replay and distillation methods

Preserving Safety During Sequential Fine-Tuning: A Factorial and Geometric Analysis of Replay

Machine Learning

Summary

Fine-tuning large language models to improve them can accidentally cause them to become less safe by responding in harmful or unwanted ways. The authors study two techniques—replaying examples of safe and unsafe prompts and copying model output patterns (distillation)—to keep safety during fine-tuning. They find that replaying examples reduces harmful responses but also makes the model refuse too many harmless requests. Distillation mainly keeps safety without causing many unnecessary refusals. Combining both methods balances safety and helpfulness better. They also show that measuring safety by how the model refuses prompts geometrically is not always reliable.

What this means in practice

  • For ai product engineers: Maintain safety alignment during sequential fine-tuning of large language models by combining replay and distillation techniques.
  • For machine learning operations teams: Deploy safer fine-tuning pipelines for code generation and summarization tasks by managing trade-offs between harmful refusal rates and over-refusals.

Authors

George Ruellan, Diana Benavides-Prado

Abstract

Benign fine-tuning can erode safety alignment in large language models, and sequential adaptation substantially compounds this risk. Prior work mitigates this erosion by replaying harmful-prompt refusals and benign-compliance examples, while other work instead matches base-model logits via distillation. However, the individual contributions of supervised exposure and distillation to safety behavior and refusal geometry during sequential adaptation remain unclear. We factor these two strategies apart and study their behavioral and geometric effects during sequential LoRA fine-tuning of Mistral-7B and Llama-3-8B across diverse task sequences (code generation, German-language summarization, and science question-answering). Behaviorally, supervised replay exposure yields the largest reduction in HarmBench attack success rate (ASR) on Mistral, averaging 51.1 points below base, but at a substantial over-refusal cost, averaging 53.4 points above base on OR-Bench-hard. Logit distillation, without any supervised replay loss, largely prevents safety erosion at a fraction of this cost: an average 5.4-point ASR reduction against only a 5.1-point rise in over-refusal. Llama shows the same asymmetry between the two strategies. Combining exposure and distillation moderates exposure's over-refusal while leaving downstream task scores largely unaffected. Geometrically, the two models diverge: on Mistral, distillation preserves the base refusal direction more closely than exposure does, whereas on Llama, refusal behavior after exposure largely survives ablation of the extracted refusal direction. This dissociation between models indicates that preserving the base refusal direction is not a reliable proxy for behavioral safety.
View PDFOpen arXiv