Beyond Direct Identifiers: Probabilistic Privacy Risk Estimation for Privacy-Conscious LLM Query Delegation
2026-08-10 • Cryptography and Security
Cryptography and SecurityComputation and Language
AI summaryⓘ
The authors study privacy risks when people talk to language models, noting that it's not just obvious personal info that can identify users but also subtle personal details. They improve a method called Privacy-Conscious Delegation (PCD) by adding a way to estimate how private a conversation is using a concept called k-anonymity. They created a new dataset with real user queries to test their approach. Their results show that this method works well with larger models like Llama-3.2-3B, balancing privacy and answer quality better than smaller models. The authors suggest k-anonymity is a helpful tool for improving privacy in these systems.
Privacy-Conscious Delegationk-anonymitypersonally-identifiable informationself-disclosurelocal language modelsprivacy-utility tradeoffPUPA-SD datasetprobabilistic estimationLlama-3.2-3Buser-LLM interactions
Authors
Li Siyan, Zhou Yu, Julia Hirschberg
Abstract
Recent work on protecting privacy during user-LLM interactions often focuses on direct, explicit identifiers: the personally-identifiable information (PII) captured by standard detectors. One such approach is Privacy-Conscious Delegation (PCD), where a local LLM acts as an intermediary. However, privacy risk does not stem solely from explicit identifiers but also PII-free self-disclosures, leaving users identifiable through combinations of quasi-identifying traits. We investigate a probabilistic variant of PCD, where we augment its objectives with an LLM-driven probabilistic estimation of k-anonymity. To facilitate this, we first create the PUPA-SD dataset, which contains naturalistic user queries with self-disclosure. Our preliminary results indicate that optimizing PAPILLON on PUPA-SD improves quality on unseen conversations across a variety of local models and produces the best privacy-utility balance for Llama-3.2-3B, while smaller models struggle to jointly optimize quality and privacy. We propose k-anonymity as a useful auxiliary metric for tackling PCD.