Ai reviewer training risks causing loss of scientific judgment diversity
When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation
Machine Learning
Summary
AI models are starting to review scientific papers, but when these AI-generated reviews are used to train new AI reviewers, the variety and richness of their judgments shrink. The authors call this problem scientific-judgment collapse. They studied this effect using several versions of AI models trained on real and synthetic reviews. To fix it, they created a system called TrustReviewer that carefully curates training data and adjusts AI behavior during review to keep diversity and quality high.
What this means in practice
- •For ai platform developers: Build AI-assisted peer review tools that maintain diverse and high-quality scientific evaluations by avoiding training on synthetic reviews that cause judgment collapse.
- •For academic conference organizers: Use curated training data and test-time correction methods to improve the reliability of AI-supported paper reviews and recommendations at conferences.
Authors
Sy-Tuyen Ho, Minghui Liu, Furong Huang
Abstract
Large language models (LLMs) increasingly participate in scientific evaluation, both as automated reviewers and as assistants to human reviewers. As model-generated reviews enter public data and future training corpora, AI peer review can become recursive: later reviewers learn from judgments produced by earlier models. We study one step of this feedback loop in a controlled setting. Starting from Llama 3.1 8B, we first fine-tune a reviewer on official ICLR reviews from 2018--2023 and then train four successor models on ICLR 2024 data with systematically varied mixtures of official and model-generated reviews. Our study shows that introducing synthetic reviews compresses rating distributions and reduces both same-paper and corpus-level semantic diversity. We call this pattern $\textbf{scientific-judgment collapse}$. To mitigate this failure mode, we introduce $\textbf{TrustReviewer}$, an open-source LLM-based system for generating peer reviews of AI and machine learning papers. TrustReviewer intervenes at two complementary stages. For training-time prevention, we train the core reviewer in a single stage on a curated corpus designed to reduce low-quality and semantically degenerate supervision. For test-time correction, paired activation steering aims to further mitigate residual tendencies toward collapsed judgments without further training or additional expert annotation. Together, these results characterize a concrete risk of recursive reviewer training and provide practical interventions for preserving judgment diversity and improving recommendation alignment in AI-assisted scientific evaluation.