Prompt template choice affects safety alignment in knowledge distillation
Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment
Computation and Language
Summary
Sometimes, computers are taught how to respond safely to tricky questions by copying a 'teacher' model to a 'student' model, a process called knowledge distillation. This study found that the way instructions (called prompt templates) are written during this copying affects how well the student model stays safe. Using chat-like instructions makes the student more likely to give unsafe replies than using more neutral instructions. The researchers tested this with three different language models and several safety tests.
What this means in practice
- •For machine learning engineers: Choose non-chat prompt templates during knowledge distillation to better preserve safe behavior in language models.
- •For ai content moderation teams: Monitor how prompt templates used for model updates may affect the system’s compliance with safety policies.
Authors
Anjila Budathoki, Manish Dhakal, Benjamin M. Ampel, Yi Ding
Abstract
Prior research has demonstrated that the choice of prompt template during Supervised Fine-Tuning (SFT) significantly impacts the robustness of safety alignment afterwards. However, the influence of template selection during Knowledge Distillation (KD) from teacher to student remains largely unexplored. Thus, we fill this gap by analyzing how different template configurations influence the pre-existing safety alignment of the student. We observe a significant degradation of safety alignment present in the aligned base instruct-tuned model. Specifically, we find that utilizing chat templates renders the model more compliant with harmful queries compared to a non-chat template. These findings are consistent across three models: LLaMA, Gemma and Qwen model families and are evaluated across multiple safety benchmarks. We further show that using a non-chat template during distillation better preserves the base student's internal representations, while chat template distillation induces a larger representational shift. Code: https://github.com/anjilab/role-of-prompt-template-in-kd