Large language models break rules in late network layers
Where Do LLMs Decide to Break the Rules? Mechanistic Localization of Prompt Injection Compliance
Cryptography and Security
Summary
Sometimes AI language models ignore their instructions if tricked by sneaky prompts. This paper finds that although the AI recognizes these bad instructions early on, it actually decides to follow them much later in its processing, near the end of its network. The researchers show that fixing the AI’s behavior at this late stage can stop the model from obeying harmful instructions. They also found that this decision-making part is quite small and similar across different AI models.
What this means in practice
- •For ai safety teams: Identify and intervene on prompt injection attacks by targeting the late-layer decision point inside language models to prevent harmful rule-breaking.
- •For security analysts: Develop tools for robust detection of prompt injection attacks using model internals from the late layers to improve resilience against obfuscated inputs.
Authors
Rui Wen, Jiayang Liu, Zeyu Yang, Jun Sakuma, Lu Sun
Abstract
When a prompt injection attack succeeds, a Large Language Model (LLM) abandons its assigned system role to comply with an adversarial instruction. While prior work has extensively quantified how often this occurs, we ask a more fundamental question: where inside the network does the model actually decide to break the rules? Using layer-by-layer causal activation patching across five models (4B to 32B parameters), we find a clear dissociation: attack information is linearly decodable from the first layer, yet causal leverage over the model's behavior is negligible until a late-layer bottleneck in the final third of the network. Patching this bottleneck reverses compliance in 77--92\% of cases. We show that the compliance mechanism occupies a compact linear subspace (rank-8 in 4B and 14B models, scaling to rank-64 at 32B) and is architecturally stable across varying model families. Finally, we validate our mechanistic account by showing that this causal peak layer is also the representationally optimal site for detecting attacks, outperforming early-layer classifiers that degrade under surface-level obfuscation such as leetspeak substitution. This alignment between causal leverage and detection performance provides converging evidence that the late-layer bottleneck captures decision-relevant computation rather than merely reflecting an artifact of the intervention.