Safe response detection fails without unsafe reference direction
A Safe Prototype Is Not a Safety Direction: Reference Dependence and Prompt Confounds in Response-Safety Embeddings
Machine LearningComputation and LanguageCryptography and Security
Summary
The paper shows that simply measuring how close a response is to a known safe example doesn't reliably tell if it's really safe. The authors found that you need to compare safe responses to unsafe ones to know which way is 'safe' in the data. They tested this idea on several datasets with known safe and unsafe responses and showed that methods using unsafe examples work much better than just using safe examples. This means that recognizing unsafe replies requires more than just knowing what safe ones look like.
What this means in practice
- •For ai safety engineers: Improve response safety detection by using both safe and unsafe examples to better identify unsafe outputs in language models.
- •For machine learning developers: Design safer response classifiers that account for direction and reference points in embedding spaces rather than relying on average safe prototypes.
Authors
Sahil Kadadekar
Abstract
Can response safety be scored by cosine similarity to the mean embedding of known-safe responses? A recent sleeper-agent detector proposes exactly this score, yet the raw positive-centroid rule is not identified: positive observations locate the safe class relative to an encoder origin, but do not determine which direction separates safe from unsafe responses. We audit the rule on two prompt-controlled, human-labeled corpora and one auxiliary jury-labeled source control, using four frozen encoders and prompt-grouped splits. On the human-labeled corpora the safe prototype reaches ROC-AUC 0.457-0.545, with two cells significantly below chance and one above, while an explicit safe-minus-unsafe reference reaches 0.588-0.738 on the same embeddings; on the jury control the prototype is inverted (0.358-0.405) and the reference reaches 0.754-0.793. At validation-calibrated 5% false-safe thresholds, the reference accepts more safe responses on PKU-SafeRLHF (0.153-0.263 versus 0.039-0.061 across encoders) and Aegis (0.189-0.291 versus 0.004-0.045), but not reliably on BeaverTails. A fully unlabeled held-out reference recovers part to most of the referenced ranking, much less when only 5% of the pool is unsafe, whereas 80-634 labeled unsafe responses recover most of it. Prompt-only ablations show that prompt-label composition can inflate uncontrolled evaluations. This is a bounded result about a raw positive centroid, not all one-class methods or safety-specialized guards. A class mean is a location, not necessarily a safety direction; a declared reference with enough unsafe mass identifies orientation.