Attention based method improves detection of indirect prompt injection attacks

Attention-based Classification for Indirect Prompt Injection

Machine Learning

Summary

Indirect prompt injection attacks sneak harmful instructions into data that language models process, which threatens how these models behave. The authors studied an existing detection method called Attention Tracker and found it struggles with realistic settings where complex tool outputs confuse the model's attention. They propose a new method, AttnDA, that uses a more detailed analysis of attention patterns and performs better at detecting attacks, especially when trained on data similar to where it will be used. However, detecting attacks in all environments remains hard, showing attention patterns alone are not always reliable indicators.

What this means in practice

  • For ai security teams: Improve monitoring tools for language model deployments by using AttnDA to better detect indirect prompt injection attacks with attention-based classifiers.
  • For llm platform engineers: Integrate learned attention-head based classifiers to increase reliability of prompt injection attack detection within agentic language model systems.

Authors

Amelia Kawasaki, Stefan Lee

Abstract

Indirect prompt injection (IPI) attacks, in which adversarial instructions are embedded in external data processed by a language model, pose a significant threat to the integrity of deployed LLM agents. Attention Tracker (Hung et al., 2025) is a recently proposed white-box detection method that identifies prompt injection attacks by monitoring how a model's attention is distributed between user task tokens and data tokens, claiming dataset-agnostic detection via a scalar focus score threshold. In this paper, we provide the first evaluation of Attention Tracker in a realistic agentic IPI setting using the AgentDojo benchmark, finding that the dataset agnosticism claim does not hold. We find that in agentic environments, long structured tool outputs subvert the fundamental assumptions Attention Tracker makes about attention that the method depends on and produces near-chance detection on AgentDojo across most model families. We propose AttnDA, an extension replacing the scalar threshold with a Fisher Linear Discriminant Analysis classifier over the per-head focus score vector, and evaluate both methods in a cross-dataset setting spanning four benchmarks. AttnDA improves detection performance across agentic and tool-integrated benchmarks, though transfer to AgentDojo remains challenging regardless of classifier choice. A covariance ablation further shows that removing inter-head correlation structure substantially contributes to performance, with complex covariance assumptions causing significant degradation on cross-domain data. These results indicate that the attentional distraction signal underlying focus-score-based detection does not transfer reliably to agentic environments, and that learned linear classifiers over head-level attention features offer improved detection performance, provided training data from the target deployment domain is available.