Large language models learn to separate instructions from data for safer use
Post-Training Large Language Models with LLMON
Artificial IntelligenceMachine Learning
Summary
Large language models currently treat all parts of their instructions and data the same way, which can cause problems like hackers tricking the model or unclear execution of commands. The authors created a new method called LLMON that forces the model to clearly separate instructions from data by using special labels. They trained large models on a big dataset with these structured inputs and found that it greatly reduces security risks without hurting performance. This approach also helps understand how the model processes instructions internally, all without changing the model design.
What this means in practice
- •For ai application developers: Build language model applications that resist prompt injection attacks by explicitly separating instructions and data without changing model architecture.
- •For security engineers: Improve security testing and evaluation of LLMs by using attention-based metrics to identify vulnerabilities and failure modes in instruction execution.
Authors
Basel Shbita, Michael Hind, Chad DeLuca, Nathan Fulton, Shubhi Asthana, Farhan Ahmed, Anisa Halimi, Yada Zhu, Mauro Martino, David Cox, Dan Gutfreund
Abstract
Large Language Models (LLMs) treat all context tokens uniformly, lacking explicit separation between executable instructions and passive data. This leads to prompt injection vulnerabilities, ambiguous instruction binding, and unreliable execution control. We present a post-training methodology using LLMON (LLM Object Notation), a framework that enables separating instructions from data and requires explicit identifier-based execution. We construct a large-scale structured training corpus of 4.6B tokens spanning general, RAG (Retrieval-Augmented Generation), and tool-calling data, augmented with distractor-infused variants in which competing instructions and irrelevant data segments are present in context but not referenced. Post-training two Granite-4.1 base models (3B and 8B) through supervised fine-tuning followed by reinforcement learning, we show that LLMON-structured supervision reduces attack success rates on indirect prompt-injection benchmarks from 53-100% to 0.02-5.6% while preserving general capabilities. We further introduce attention-based interpretability metrics that diagnose failure modes at individual transformer layers for structured inputs, revealing how our approach reshapes internal attention patterns. Our results demonstrate that explicit execution semantics can be taught through standard post-training without architectural modification.