When ai companies should tell police about private chats
When Is Disclosing Private Conversations with AI to the State Justified? Proactive Referral Policies of Major U.S. AI Companies through the Lens of Contextual Integrity
Computers and Society
Summary
Sometimes, AI companies tell the police about user chats that seem dangerous without being asked. This can help stop violence but might also cause problems by exposing people who just shared scary thoughts with AI. The authors studied when this is okay and found that no major AI company clearly explains their rules or lets users know when this happens. They suggest better policies that clearly set limits and tell users in advance.
What this means in practice
- •For ai company policy makers: Create clearer public policies that define when and how to share user conversations with law enforcement as a rare exception.
- •For privacy compliance teams: Assess and update disclosure practices to align with accepted norms of confidentiality and user notification after proactive referrals.
A position paper. It proposes an approach and reports no results.
Authors
Minjong Cheon, Hanseon Joo, Ook Lee, Jongchang Ahn
Abstract
AI companies engage in "proactive referral": notifying law enforcement, without a government request, of user conversations they judge dangerous. This poses a dilemma: referral may expose people who merely confided dangerous thoughts to an AI to investigation, while not referring may leave actual violence unprevented. This study asks under what conditions proactive referral can be justified and to what extent companies' public policies disclose those conditions. Drawing on contextual integrity theory and the norms governing exceptional information sharing in confidential counseling, online posts, and private communications, the study develops eight justification criteria classified as required, presumptive, or recommended. Three evaluators analyzed 27 public documents from six major U.S. AI companies using the criteria applicable to corporate policies. No company clarified whether broad sharing provisions or narrower safety notices took precedence, and assessments of referral scope depended on which provisions were given precedence. No company disclosed the specific factual thresholds and exception conditions governing referral decisions, or whether users would be notified afterward. The study turns contextual integrity into a stage-by-stage framework for proactive referral, shows that policy documents must be read in light of their relationships, and proposes policy elements for keeping proactive referral a narrow exception disclosed in advance.