Context changes test large language models on sharing private information
ConCI: A Controlled Counterfactual Benchmark for Contextual Integrity in LLMs
Artificial Intelligence
Summary
People share personal information differently depending on the situation, but existing tests for AI models don’t clearly show if models understand this context or just say no to sharing. The authors created ConCI, a special test where each scenario is paired with two others that change only one thing at a time, making it easier to see if models really get the context. They found that even the best AI models often make mistakes, especially by not recognizing when it’s okay to share information after the situation changes. Bigger models improve but still struggle, and asking models to explain their answers helps only a little.
What this means in practice
- •For ai safety engineers: Evaluate large language models’ understanding of when sharing personal data is appropriate or not under different scenarios.
- •For privacy compliance teams: Test AI systems for context-sensitive data sharing to reduce risks of inappropriate personal information disclosure.
Authors
Juhyeon Park, Sumin Yu, Taesup Moon
Abstract
Contextual integrity (CI) holds that the appropriateness of sharing personal information depends on the context of the information flow, not on the information alone. However, existing CI benchmarks often use independent scenarios that vary multiple CI parameters simultaneously and are dominated by inappropriate flows, making it unclear whether correct predictions reflect sensitivity to context or a general refusal tendency. We present ConCI, a controlled-counterfactual benchmark in which an anchor flow is paired with two counterfactuals that each modify a single CI parameter. The resulting triplet is scored jointly, with credit given only when all three flows are judged correctly. Built from six existing benchmarks, ConCI contains 1,592 flows in 556 triplets, is approximately label-balanced, and is validated by seven human annotators per flow. Across seven LLMs, many correct anchor predictions fail on at least one counterfactual, and even the strongest model errs on about a fifth of triplets. Furthermore, models perform near ceiling on inappropriate flows but weaker on appropriate ones, often continuing to judge a flow inappropriate even when a contextual change should make it appropriate. Larger models narrow this bias toward non-disclosure but do not eliminate it, and reasoning yields only modest gains.