Human and AI interrogators differ in reading Turing Test signals
Beyond pass or fail: an information-processing account of the Turing Test
Human-Computer Interaction
Summary
The Turing Test checks if a machine can mimic a human well enough to fool a person. This paper shows that failing the test can mean two different things: the machine's replies might be too similar to human replies with no telltale signals, or the person judging might miss those signals. The researchers put humans and AI (GPT-4 and GPT-5) in both roles of judge and responder, and found that humans struggled more to detect AI answers as machines with GPT-5’s improvements. Meanwhile, AI judges got better at spotting differences, though their explanations didn't fully match how they decided. This shows humans and AI use different ways to detect who’s behind the messages.
What this means in practice
- •For ai system developers: Improve AI evaluation tools by incorporating both encoding quality and interrogator readout capacity to better detect human-likeness in AI responses.
- •For security analysts: Enhance detection methods for AI-generated content by designing tools that consider differences in human and AI judgment processes shown in this study.
Authors
Qiwei Zhao, Oriana Pansardi, James W. Strachan, Eugenio Scaliti, Guido Manzi, Giulia Borghini, Fabio Manzi, Alessandro Rufo, Frédérique de Vignemont, Michael S. A. Graziano, Stefano Panzeri, Cristina Becchio
Abstract
The Turing Test is passed when a human interrogator cannot distinguish machine-generated responses from human ones. Yet behavioral indistinguishability can reflect two fundamentally different failures: a failure of encoding, in which machine and human output distributions overlap so closely that source information is effectively absent; or a failure of readout, in which source information is present but the interrogator fails to extract it. To dissociate these possibilities, we inverted the Turing Test, placing humans and AI systems in symmetrical interrogator roles. Using a quantitative information-processing framework, we separated encoding, the information contained in the text; readout, the extraction of that information by an interrogator; and meta-readout, the interrogator's explicit access to the features guiding its own judgments. Across two series of experiments involving more than 3800 human interrogators and two generations of GPT models (GPT-4 and GPT-5), we found a striking dissociation across processing levels and interrogator types. As AI-generated responses became more human-like from GPT-4 to GPT-5, source information became more difficult for human interrogators to extract. GPT-5 interrogators, by contrast, read out source-discriminating information more accurately than GPT-4 interrogators. This improvement, however, came with a specific limitation: GPT-5 explanations only partially aligned with the feature weightings that actually predicted their judgments. Human interrogators showed the complementary pattern, with modest readout alignment but comparatively higher meta-readout alignment. Together, these findings suggest a double dissociation between readout and meta-readout across human and GPT interrogators.