Training agents to hide private facts from observers using twin worlds
Indistinguishable Agents: Training Agent Privacy as Twin-World Indistinguishability
Machine LearningComputation and Language
Summary
Many AI agents help users by searching or messaging, but what they do can reveal private information. The authors propose training these agents with a method that compares two almost identical situations that only differ in the private fact. By making it hard to tell which situation the agent is in, this approach keeps private facts hidden better than previous methods. They tested this on tasks like shopping and messaging and found it prevents leaks without stopping the agent from working well.
What this means in practice
- •For ai product developers: Create AI assistants that complete tasks while better protecting users' private facts from being inferred by observers.$Commercial implications: This enables developing privacy-preserving AI agents that attract privacy-conscious consumers and comply with data protection regulations.
- •For privacy engineers: Design and evaluate agent behaviors that minimize information leakage about private user data beyond direct mentions.
Authors
Jiaming He, Guanyu Hou, Yi Yu, Xudong Jiang
Abstract
LLM agents search, shop and send messages for their users, and every action is seen by some party. Existing privacy training rewards agents for not mentioning protected items, yet an observer can infer a private fact from what the agent does. Since an observer learns a fact only if its view changes with the fact, we propose \methodfull (\method), which trains on twin worlds that differ only in the private fact. A frozen distinguisher scores each observer's view by its log-likelihood ratio for the true world, and paired GRPO trains the agent on task success minus this ratio. We prove that this objective controls the Bayes success of every observer on every channel. On shopping and messaging tasks \method removes the leakage that disclosure judges and query-level rewards miss, while the agent still completes its tasks.