Detecting who contributed scientific review insights beyond text authorship
Who Wrote It Is Not Enough: Detecting Who Contributed the Insight
Computation and Language
Summary
It is often hard to tell who truly contributed the ideas behind scientific reviews, especially when AI helps write the text. The authors created a dataset and methods to identify if insights come from humans, AI, or both, looking beyond just who wrote the words. They found that human insights tend to add outside knowledge and judgment, while AI insights usually stick close to the original paper. This means that even if AI rewrites text, the origin of the ideas can still be detected.
What this means in practice
- •For peer review platforms: Identify whether scientific review comments derive from human, AI, or combined contributions to improve transparency.
- •For academic publishers: Enhance editorial processes by detecting the true origin of review insights, distinguishing between human and AI input.
Authors
Zhuoyang Zou, Abolfazl Ansari, Jiaxi Yang, Delvin Ce Zhang, Qian Chen, Dongwon Lee, Wenpeng Yin
Abstract
As LLMs increasingly assist scientific writing and peer review, detecting who wrote the text is no longer sufficient: we need to determine who contributed the underlying insight. We introduce Insight Provenance, the task of identifying whether a review insight originates from a human, an LLM, or their hybrid contribution. We construct InsightProv-v0 from 4,057 scientific papers and 12,660 human reviews, simulating different levels of LLM involvement with GPT-4o, Gemini, and DeepSeek and annotating provenance at the sentence level. We show that strong performance on raw data can be misleading, as models exploit linguistic and textual-authorship shortcuts that degrade substantially under progressively debiased evaluation. We therefore propose a two-stage adversarial framework that suppresses shortcut signals while preserving provenance-relevant information. Beyond detection, extensive analyses reveal what makes intellectual authorship identifiable: paper grounding and neighboring review context provide complementary provenance signals, while human, hybrid, and AI insights systematically differ in their information sources and failure modes. Most strikingly, AI insights predominantly remain close to generic or paper-provided information, whereas human insights more often introduce external knowledge and independent judgment. These findings suggest that while wording can be rewritten by an LLM, the provenance of an idea leaves a deeper and more persistent signal.