Llms confuse who says what enabling fake role attacks
Understanding and Mitigating Role Spoofing in LLMs
Cryptography and SecurityArtificial Intelligence
Summary
Large language models (LLMs) can be tricked into mixing up who said what during a conversation. This mix-up can let attackers pretend to be someone else by changing style and labels in the text. The authors show that models rely on both how text looks and what it says to track speakers, and attackers can exploit both. They test a training method that helps the model ignore misleading content but not misleading text structure, explaining why newer models still fail. Finally, they introduce a way to detect these impersonation attacks effectively.
What this means in practice
- •For ai security engineers: Detect role spoofing attacks in conversational AI by using model probes trained on role prediction to improve system safety.
- •For chatbot developers: Reduce impersonation attacks in chatbots by implementing adversarial training that targets content-based spoofing vulnerabilities.
Authors
Ezra Edelman, Benjamin Edelman, Surbhi Goel, Eran Malach
Abstract
Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we call role spoofing}. In this work, we seek to understand and mitigate role spoofing. We study how models track "who said what" in a conversation, showing that they use cues from both content (e.g., semantics and writing style) and structure (e.g., control tokens and formatting). We show that LLMs are highly susceptible to role spoofing attacks that combine both fake structure and content. Additionally, in a toy setting, we show that adversarial training which teaches the model to ignore content-based injections does mitigate content-based attacks but leaves the model susceptible to structure-based attacks. This can explain why frontier models, with increased resistance to many injection attacks, can still be susceptible to role spoofing. Finally, we propose a new method for detecting role spoofing using linear probes trained to predict roles in a conversation, and show that it is effective across different settings.