Provenance aware transformers separate trusted sources in language models
Origin Is All You Need: Provenance-Aware Transformers for Structural Trust-Boundary Separation
Cryptography and Security
Summary
Large language models sometimes get tricked by confusing instructions that mix trusted info with user data. The authors designed a new type of transformer model that knows where each piece of information comes from by tagging it and treating it differently. This helps the model keep trusted instructions separate from untrusted inputs, making it safer to use. Their method works well without losing the model’s ability to do its tasks.
What this means in practice
- •For ai system engineers: Build language models that resist harmful prompt manipulations by structurally separating trusted and untrusted inputs.
- •For security teams: Enhance the safety of AI conversational platforms by embedding source tracking to prevent indirect prompt attacks.
Authors
Yuxuan Zhang, Jeff Huang, Guofei Gu
Abstract
Indirect prompt injection (IPI) remains a central safety and security challenge for large language model (LLM) systems because standard transformers lack architectural notion of source authority. Retrieved documents, user inputs, and system instructions are all processed through the same undifferentiated attention mechanism, forcing the model to infer from wording alone what should be obeyed and what should be treated as data. We propose Provenance-Aware Transformers, a provenance-aware defense that makes application-supplied source labels actionable inside the model. Each input token is assigned a ring ID encoding its origin, and the model is augmented with origin embeddings, a learnable origin attention bias, and a learnable origin scale that preserves provenance under normalization. The resulting architecture enforces a structural boundary between authoritative and non-authoritative sources during generation. To instantiate this architecture on released pretrained models, we propose a two-stage fine-tuning pipeline to teach the model origin semantics and task behavior under ring constraints. Evaluation shows that Provenance-Aware Transformers maintain robust resistance to IPI both in-distribution and out-of-distribution while preserving utility comparable to the base pretrained model. More broadly, our work shows that exposing provenance as a first-class architectural signal can shift LLM safety alignment from brittle pattern matching toward explicit trust separation.