New method measures outside info in peer reviews using language models
How Much Were You Told? Measuring External Information in Peer Reviews
Computation and Language
Summary
It is hard to tell if peer reviews used outside help like large language models to write the critique or just polished their own words. The authors created a method called Self-Conditioning that looks at how much extra information a review contains beyond the paper and instructions, rather than just how it sounds. Their method can very accurately spot reviews that were fully outsourced to AI versus lightly edited ones. It also tracks how much outside info the AI used and is harder to fool by simple rewriting tricks.
What this means in practice
- •For conference program committees: Detect when peer reviews rely heavily on external AI-generated information instead of the reviewer’s own critique to ensure review integrity.
- •For academic integrity teams: Evaluate submitted reviews to identify cases of fully delegated AI-written critiques versus reviews polished by AI to maintain fairness in evaluation.
Authors
Matthieu Dubois, Pablo Piantanida, François Yvon
Abstract
Conference policies distinguish using Large Language Models (LLMs) to polish one's own review from delegating the critique, but current Artificial Text Detection (ATD) methods largely measure surface form rather than the origin of its content. We instead measure the external information carried by a review: information not explained by the reviewed paper and a generic reviewing instruction. We propose Self-Conditioning, an unsupervised information-theoretic estimator that compares the likelihood of a review under its production context with its likelihood when that context is augmented with hints extracted from the review itself. On the IntelLabs peer-review benchmark, Self-Conditioning separates fully-delegated from machine-polished reviews with AUC up to $1.0$ while remaining largely insensitive to surface rewriting. Moreover, as generators receive increasing amounts of externally-provided information, their scores move monotonically towards the human regime, unlike standard ATD baselines. High-temperature sampling can evade the estimator, but at the cost of output quality.