Channel-aware training cuts prompt injection attack success rates drastically
TRACER: Trace-Regularized Training Against Indirect Prompt Injection
Machine Learning
Summary
Large language models can be tricked when untrusted information is sneaked into their prompts, causing them to give wrong or harmful answers. The authors show that these attacks work differently depending on how the information is sent to the model, with some ways being more vulnerable than others. They propose a new training method called TRACER that teaches models to recognize and resist these trick messages no matter how they appear. Tests show TRACER greatly reduces the success of these attacks while keeping the model’s normal accuracy mostly intact.
What this means in practice
- •For ai product teams: Develop safer language model applications by training models to resist prompt injections across different data input methods.
- •For security engineers: Improve defenses against indirect prompt-based attacks by deploying channel-aware adversarial training techniques like TRACER.
Authors
Kshitij Mishra, Salem Lahlou, Nils Lukas
Abstract
Indirect prompt injection uses untrusted retrieved content, tool outputs, or agent messages to override an LLM's intended behavior. We show that defense robustness is channel-conditioned: the same attack succeeds 18% of the time as a follow-up message but 53% when placed in retrieved context. We propose TRACER (Trace-Regularized Adversarial Channel Exposure with Reward-shaping), a channel-aware adversarial training method that samples presentation wrappers and combines an answer-anchored, evidence-gated reward with ATRC, which aligns attacked reasoning with clean evidence when an injection is invalid. On GSM8K with Qwen2.5-3B-Instruct, TRACER reduces RAG-framed ASR from 65.5% to 6.1% while preserving 78-82% clean accuracy. Joint RAG+tool training achieves 13.9% average ASR across framings at 3B scale, while a single epoch at 7B reaches 9.4%. These results demonstrate that robust defenses must explicitly train and evaluate across deployment channels.