Linear probes detect harmful prompts well despite model changes
Linear Probes Transfer from Base to Distilled Language Models despite Rotation
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Linear probes are simple tools that check inside language models to spot harmful requests. The authors tested if probes trained on one model still work well on newer, changed models without retraining. They found probes still perform well even after the models were distilled or instruction-tuned, despite some internal rotations. This means auditors can reuse probes across model versions, saving effort. However, probes sometimes flag safe prompts as harmful based on wording, not true harm.
What this means in practice
- •For ai safety teams: Reuse tested harmful content detectors on updated language models without retraining probes each time.
- •For content moderation engineers: Deploy classifiers that read model internals for better harmful prompt detection beyond just text cues.
Authors
Marios Tsatsos
Abstract
Linear probes, classifiers that read a language model's internal activations, are widely used to detect harmful requests. They are cheap to train and run, and an auditor who has trained one would like to reuse it when the model is replaced by a successor rather than retrain it for every release. We test how well probes can be reused. For four base models (Qwen2.5-Math-7B, Llama-3.1-8B, Qwen2.5-14B, Qwen2.5-32B) we train a probe on the base model with the XSTest prompt set and apply it, unchanged, to two successors of that base: the DeepSeek-R1 distilled reasoning model and the developer's instruction-tuned chat model. We find that the successors rotate the direction separating unsafe from safe prompts by 17$^\circ$ to 35$^\circ$, yet on XSTest the transferred probe's AUC changes by only $-0.011$ to $+0.027$, and on an independent OR-Bench evaluation, never used for training, by $-0.015$ to $+0.053$. On XSTest, whose safe prompts deliberately contain alarming words, the probes beat a classifier that sees only the prompt text by 0.15 to 0.20 AUC. On the OR-Bench prompts that margin shrinks to 0.03 to 0.20, and the probes flag 29\% to 42\% of prompts that sound harmful but are benign, so part of what they read is the appearance of harm rather than harm itself. A probe trained on a base model can thus be reused, in these families, after distillation and after instruction tuning; how much it reads beyond textual cues depends on the test set.