Confidence scores help spot hijacked AI labels but often miss attacks
Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities
Computation and LanguageCryptography and Security
Summary
Some programs use language AI models to pick labels based on their top guess. Bad actors can sneak in tricky text that shifts the label to whatever they want, called a hijack. The authors tested a way to catch these hijacks by watching the model’s confidence score: low confidence might mean a hijack. This method worked fairly well at ranking suspicious cases but missed many real hijacks, especially when attackers cleverly claimed certainty. So, just checking confidence isn’t enough to stop all AI label hijacks.
What this means in practice
- •For machine learning engineers: Use confidence-based monitors to help detect some label hijacks in AI classifiers but remain aware of high miss rates with confident attacks.
- •For ai security teams: Test AI systems’ resistance to prompt-injection attacks using confidence scores as one alarm signal while preparing for sophisticated bypasses.
Authors
Junwei Yao, Jianan Liu, Yongxi Zhou, Zike Yang
Abstract
Programs that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker chooses, a hijack. A monitor that does not know the attacker's label can read the model's own confidence and raise an alarm when the top probability is low. We test this monitor on AG News topic classification with five systems, three of them served through a provider's API. Pooled over five families of injected text, low confidence ranks hijacked answers above clean ones on four of the five systems (AUROC 0.86-0.90). Ranking them does not stop many of them. A threshold set so that at most 5% of clean answers to separate calibration articles alarm falls among the least confident clean answers, and every more confident hijack passes. In our main gate run on the three API-served systems, the monitor misses 51-70% of the hijacks caused by a plain injected note (per-system point estimates). When the note adds one handwritten sentence that calls the target certain and asks for full probability, the monitor misses 99-100% of the hijacks, although low confidence still ranks them at AUROC 0.70-0.79. Confidence monitors should be tested at their operating alarm rate, against attackers who claim certainty.