Large language model agents leak private data indirectly to users
InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants
Artificial Intelligence
Summary
When language-based assistants do everyday tasks for users, they might accidentally reveal private information not by outright saying it, but by giving hints others can use to figure it out. The authors created a test called InferPriv to see how much private info could be guessed from these assistants' replies. They found that even when these assistants avoid saying secrets directly, a lot of private details can still leak through hints. This means checking only for obvious leaks misses some important privacy risks.
What this means in practice
- •For ai product teams: Evaluate and reduce indirect privacy leaks in AI assistants that handle sensitive user data during third-party communication.
- •For privacy compliance officers: Assess inferential privacy risks in LLM-based systems to enhance data protection policies and avoid subtle information exposure.
Authors
Pengrun Huang, Yiwei Fu, Luoxi Meng, Kamalika Chaudhuri
Abstract
Large language model (LLM)-based personal agents are increasingly deployed to handle everyday tasks on users' behalf. These agents have access to private user information, while their tasks involve communicating with third parties; this raises a critical privacy concern that agents may inappropriately reveal sensitive user information to these recipients. Existing benchmarks primarily assess direct privacy leakage, where an agent explicitly states a pre-specified secret or attribute. However, they do not capture \textit{inferential leakage}, where the model leaves indirect cues in the reply that implicitly leaks sensitive attributes. We introduce \textsc{InferPriv}, a benchmark that measures whether an honest-but-curious recipient can infer a user's sensitive attributes from an LLM's reply on routine task. Evaluating five frontier models, we find that inferential leakage substantially exceeds direct leakage across models and prompts. Under a privacy-aware system prompt, GPT-6.1-Sol, Sonnet-5.5, and Gemini-3.8-Flash achieve near-zero direct leakage, yet the inferential leakage rate remains between 16.2--25.2\%. These findings suggest that evaluating explicit leakage alone can substantially understate the privacy risks.