AI writing detection depends on context and social norms

Contextuality and Contradictions: Exploring Real-World Identifications of AI-Generated Writing

Computation and Language

Summary

People often try to tell if a text was written by AI or a human, but the ways they do this are not straightforward or consistent. The authors studied how Wikipedia editors and students decide if writing is AI-generated and found that their judgments rely on many factors, including the style, content, and assumptions about the writer. Instead of just spotting AI-like features, people focus on when something seems out of place or inappropriate for the situation. Tools that claim to detect AI writing without considering context may misunderstand these social rules and cause problems.

What this means in practice

Authors

Michelle Morgenstern, Julia Witte Zimmerman, Tabia Tanzin Prama, Kristine Harootunian, June Clark, Bryn Loftness, Calla Beauregard, Juniper Lovato

Abstract

As AI-generated content spreads into nearly every domain of human interaction, practices and discourses of AI detection, both formal and informal, have become an everyday part of how people interact with text and with each other. In this paper, we explore the real-world practices and discourses of AI identification that have emerged in response to a shifting discursive landscape where the human provenance of a text can no longer be taken for granted. We pair qualitative coding with computational analysis of lexical change, grounded in linguistic anthropological perspectives on language and communication, to investigate AI identifications among English Wikipedia editors and students. We find that these identifications draw on a diversity of evidentiary types, from discrete textual features and judgments about content to circular sense-making claims and presumptions about the author's moral character, and that they are contradictory, dynamic, and contextually dependent. In practice, what often motivates an AI identification is not the presence of any specific textual element, but pragmatic infelicity: a perceived mismatch between the elements present and those deemed appropriate within the text's particular context. Consequently, real-world identifications of AI-generated writing do not merely neutrally point out the perceived presence of "AI-like" features, but can be leveraged as a discursive resource to maintain and define sociolinguistic norms: what can be said in a given context, what is appropriate, and, ultimately, what is human. Further, the design and implementation of systems and tools that treat AI-likeness as a context-free property of text may therefore inherit, and enforce at scale, the norms of whichever contexts they were trained on.