Subjectivity helps detect ai generated peer reviews with high accuracy
SAGE (Subjectivity-Aware Generation Estimator): A Linguistic Framework for AI-Generated Review Detection *
Computation and Language
Summary
It can be hard to tell if a peer review was written by a human or an AI. The authors show that focusing on how people express opinions and intentions—things like subjective language and speech acts—can help spot AI-written reviews. They built a simple tool that uses these language features and got very accurate results. Combining it with existing AI detectors makes it even better at finding fake reviews. This approach is easy to understand and can improve current detection methods.
What this means in practice
- •For journal editors: Automatically screen peer reviews to identify AI-generated submissions that may lack genuine human subjectivity.
- •For content moderation teams: Enhance AI text detection tools by integrating subjectivity-aware features to better catch AI-generated reviews in online platforms.
Authors
Gonzalo Freijedo Aduna, Charles Yaiche, Patrick Bas, Alda Mari
Abstract
We propose a simple, linguistically motivated approach to detecting AI-generated peer reviews based on the inherently human trait of subjectivity. We capture subjectivity at two levels: the semantic level of evaluative language and modality and the pragmatic level of speech acts. Using features derived from speech act annotations and model-predicted subjectivity scores, we train a lightweight classifier that achieves an AUROC of 0.9566 with only these linguistic features. Combining its output with the state-of-the-art detector Sem-Detect raises the TPR at 1% FPR of Sem-Detect on the APRDB benchmark from 0.874 to 0.985, outperforming the current state of the art. Our results show that linguistic features of subjectivity provide an explainable source of information for detecting AI-generated peer reviews and can be readily integrated into existing detection methods. The code and models are publicly available on GitHub (https://github.com/Arys02/generated\_ text\_detection\_hub) and Hugging Face (https://huggingface.co/Arys02), respectively.