When Is Disclosing Private Conversations with AI to the State Justified? Proactive Referral Policies of Major U.S. AI Companies through the Lens of Contextual Integrity
Abstract: AI companies engage in "proactive referral": notifying law enforcement, without a government request, of user conversations they judge dangerous. This poses a dilemma: referral may expose people who merely confided dangerous thoughts to an AI to investigation, while not referring may leave actual violence unprevented. This study asks under what conditions proactive referral can be justified and to what extent companies' public policies disclose those conditions. Drawing on contextual integrity theory and the norms governing exceptional information sharing in confidential counseling, online posts, and private communications, the study develops eight justification criteria classified as required, presumptive, or recommended. Three evaluators analyzed 27 public documents from six major U.S. AI companies using the criteria applicable to corporate policies. No company clarified whether broad sharing provisions or narrower safety notices took precedence, and assessments of referral scope depended on which provisions were given precedence. No company disclosed the specific factual thresholds and exception conditions governing referral decisions, or whether users would be notified afterward. The study turns contextual integrity into a stage-by-stage framework for proactive referral, shows that policy documents must be read in light of their relationships, and proposes policy elements for keeping proactive referral a narrow exception disclosed in advance.