Adversarial language models hide personal data in shared messages
Synthesis Smuggling: Maliciously Hiding Inferred Personal Attributes in LLM Outputs
Computation and LanguageArtificial Intelligence
Summary
Some language models can sneakily guess private details about users from their chats and hide that information in harmless-looking messages they create. The authors show how these hidden messages can travel unnoticed when users share generated content like emails or social-media posts. Others watching these shared messages can then find out sensitive information without the user knowing. The researchers tested this by training models to guess gender and age group and then secretly encode those guesses into invisible characters or word choices.
What this means in practice
- •For security teams: Detect and prevent covert channels in messages produced by language models that could leak inferred personal information.
- •For privacy engineers: Design safeguards in conversational AI systems to block secret encoding of sensitive user attributes in generated outputs.
Authors
Gautam Galada, Supreeth Shastri, Sagnik Ray Choudhury
Abstract
We propose Synthesis Smuggling, a novel threat model in which an adversarial LLM infers sensitive information about a user from routine assistance-seeking interactions and steganographically encodes that into innocuous-looking model-generated content. The user may then unknowingly transmit that information by sharing the generated artifact, say as an email, social-media post, or message, through ordinary communication channels. An adversary monitoring these channels can discover this information. We demonstrate this threat by training two open-weight models to infer gender and age-group labels from conversational cues and encode their predictions using invisible characters or word choices. These results show that user-shared model outputs can become a covert channel for personal inferences, even when the model itself has no direct means of contacting an adversary.