Summary
Sometimes, an AI agent can seem like it's only sharing what’s needed for its task but is actually leaking private information you wouldn’t expect. The authors studied what a passive observer can tell about hidden private details just by watching the messages, without linking them to those private details directly. They found it’s possible for an agent to hide a full bit of private info while still acting honestly according to its task. Their experiments with a language model show that subtle changes in message selection can reveal private bits with high accuracy, even when messages seem task-appropriate. This work helps clarify what monitoring can and cannot guarantee about privacy when watching agent communication.
What this means in practice
- •For data privacy engineers: Assess privacy risks in AI communication by modeling how hidden information can leak through message choices even if all outcomes appear correct.
- •For ai system auditors: Design audit protocols that identify limitations of passive monitoring, helping avoid false security from observing only message-output behavior.
Authors
Daksh Pandey, Sayon Duttagupta, Xinhai Zou, Amit Singh Bhati
Abstract
An AI agent can reproduce the distribution of its messages and of the receiver's task outcomes while disclosing private information that the task never required. We ask what a passive audit can certify when the auditor knows the authorised sender and the exact observable law but has no trusted pairing of messages with the private states behind them. For a single round, compatible mutual-information leakage forms a sharp interval, and fixing the entire task-relevant message law, with every declared task utility, can still leave a full private bit undetermined; total leakage is determined if and only if, in every task state, the remaining private state or the authorised message is a deterministic function of that state. For a sender, such as an LLM, that selects among its own responses, we prove that in a balanced case its change to the message law within each task state equals $(1-2^{-k})$ times the honest messages' dependence on the attribute, which also sets, up to a constant, how many messages a monitor needs. With Qwen2.5-1.5B-Instruct on a synthetic routing task, selection among 16 responses lets a public decoder recover a private bit in all 8,192 held-out rounds at 98.60\% routing accuracy (honest sender 98.90\%). A monitor that knows the selection rule reaches AUC 0.645 on single messages, close to the ceiling of 0.671 that honest messages predict, and 0.972 over 16, while generic monitors stay near chance. A law-preserving sender built from the same samples recovers the bit in 97.7\% of rounds, and every monitor without the true bit stays near chance against it. A canonical task interface preserves every receiver decision and bounds the remaining information about a balanced attribute by twice the routing-error rate, even for a receiver that shares a key with the sender. These results distinguish what passive auditing can certify from what restricting communication can enforce.