Internal signals reveal when prompt injections succeed in AI models
Are You Reading the Right Thing? Rethinking Internal Signals for Prompt-Injection Detection
Artificial Intelligence
Summary
Prompt injections try to trick AI by adding sneaky instructions inside their inputs. The authors studied five ways to detect these sneaky injections by looking inside large AI models as they respond. Some detection methods notice the presence of injections, while others spot when injections actually succeed in changing the AI’s output. They found that methods analyzing signals closer to the AI’s final answer work better at telling when injections succeed. This insight can help improve safety checks for AI systems.
What this means in practice
- •For ai safety engineers: Improve tools that monitor language models to detect successful prompt injections and reduce harmful outputs.
- •For ai platform operators: Use better internal signal detection methods to flag compromised prompts and protect AI services from manipulation.
Authors
Sarthak Gupta, Priyanka Tembey
Abstract
Several recent studies propose detecting prompt injections from a language model's internal state. However, these methods do not measure a single security property. Residual-stream probes, attention scores, role projections, exposure probes, and concept readouts may capture injection presence, recognition, information routing, or behavioral influence. It is therefore unclear what differences in prompt-injection detection performance reveal about the internal signals these methods read. We study how five representation-reading methods vary under a common structured indirect prompt-injection setup on Qwen3.6-35B-A3B and Gemma-4-26B-A4B-it. We evaluate each method by how well it distinguishes injected prompts from benign controls. We then further analyze each detection method's ability to detect attacks in cases where they are successful and in cases where they are not. This helps us better assess what properties each detection method might rely on. Methods trained to detect injection presence or attack-related concepts generally separate unsuccessful injections more strongly from benign inputs, whereas attention diversion, exposure, and role-confusion-based methods separate successful injections more strongly on both models. Attention Tracker provides the strongest consistent successful injection separation, reaching 0.949 AUROC on Qwen and 0.887 on Gemma. These results suggest that signals closer to response formation may better encode information about an attack's success, and that detection performance can depend strongly on the prompt template. We share the data associated with this study at https://anonymous.4open.science/r/paper-evaluation-data-7877