Guard models give wrong high confidence answers when unsure
Guard Models Are Overconfident Where Base Models Are Uncertain
Machine LearningArtificial Intelligence
Summary
Guard models are designed to spot unsafe or harmful inputs and warn when something could go wrong. This paper finds that while these models usually give sensible confidence scores on normal inputs, they become overconfident and make mistakes when faced with tricky or adversarial inputs. Interestingly, the original language model they watch over still shows uncertainty in those cases. The researchers also looked inside the models and found differences in how the guard and base models process information, especially in the later stages.
What this means in practice
- •For moderation tool developers: Improve safety moderation by monitoring base language model uncertainty signals alongside guard model confidence to better detect harmful inputs.
- •For security engineers: Design defenses that account for guard models’ overconfidence in adversarial scenarios by analyzing representation differences in late model layers.
Authors
Jonghyun Hong, MinJae Jung, Minwoo Kim
Abstract
Guard models are used as safety classifiers, with confidence scores driving downstream moderation decisions. We evaluate five guard models for prompt classification and find that although several are nearly calibrated on clean inputs, adversarial attacks degrade their calibration by an order of magnitude, turning false negatives into high-confidence errors indistinguishable from correct detections. Comparing each guard with its corresponding base LM, we find that uncertainty signals often remain available, with the base model typically expressing uncertainty on the same inputs where the guard fails. Layer-wise analyses localize this guard-base divergence to later layers, where guard models exhibit sharper safe/unsafe separation and lower-rank representations, while adversarial harmful inputs lie closer to the clean-safe region. These findings highlight a mismatch between guard confidence and base model uncertainty under attack.