Papers for

academic conference organizers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Ai models vary widely in spotting errors in scientific paper reviews

Auditing AI peer reviewers: a benchmark on real scientific papers

Abstract: LLM-based peer review is now used at production scale, but its evaluation remains limited. We introduce a benchmark that injects objective, localisable errors of four types (general issues, mathematics, numerics, citations) at three severity levels into real papers and measures each reviewer's detection rate by error type and severity, together with its false-positive rate on clean copies. We also present Skepthical, a multi-agent reviewer that verifies citation-backed claims by retrieving the cited papers, mathematics with computer algebra, and numerical claims with executed code. We apply the benchmark to two corpora of 15 recent arXiv preprints each, one in astrophysics and one in machine learning (ML), injecting a total of 357 errors, and evaluate twelve reviewer systems: six single-LLM reviewers, two Claude Code configurations, three publicly available reviewers, and Skepthical. The choice of model has the largest impact on detection rate: given the same prompt, three frontier LLMs differ by about 50 percentage points in total detection on both corpora. Tools and scaffolding help when they contribute what the model cannot supply on its own. In particular, on astrophysics, Claude Code improves on a single Claude call in mathematics and numerics detection, and enforced citation retrieval gives Skepthical the highest citation error detection rate of all systems by a large margin. This citation lead, however, is not observed on ML, where the models appear to know the literature better and Skepthical's citation detection rate is statistically indistinguishable from that of the best single LLM. False-positive rates are below 10% for most systems, but vary with prompt, model, and scaffolding, with mathematics flags the main source of false positives. The evaluation pipeline, the injected errors, and the adjudication logs are released as a benchmark.

Thu 12 NovComputation and LanguageArtificial IntelligenceDigital Libraries
The gist
Checking scientific papers carefully for mistakes is a big job that some AI tools now help with. The authors created a test with real scientific papers that had made-up errors added, to see how well different AI reviewers could find those mistakes. They found that the choice of AI model makes a big difference—some models find many errors while others miss half. They also made a special AI reviewer called Skepthical that checks claims by looking up citations and verifying math and numbers, which worked especially well for spotting citation problems in physics papers. Most AI reviewers rarely say there’s a mistake when there isn’t one, but math errors are the trickiest for them.
Open → 2611.17052v1

Auto researchers struggle to match human research choices in ai studies

Can Frontier Auto Researchers Reconstruct Human Research Decisions in AI?

Abstract: Scientific research has long been driven by substantial human effort, from reviewing prior work and identifying research gaps to designing specific solutions. Recent advances in harness-based agent systems have given rise to increasingly capable Auto Researchers, which can autonomously carry out these tasks and, increasingly, entire end-to-end research workflows. Yet despite their rapidly growing capabilities, how closely the research decisions and outputs produced by auto researchers align with those of human experts remains underexplored. We investigate this gap through the lens of research reconstruction and study: Given comparable research context, how similar are the research decisions and outputs produced by Auto Researchers to those of human researchers in AI research? To answer this question, we introduce AutoResearcher-Human Research Similarity (AutoHRS), a stage-wise evaluation framework for assessing how faithfully auto researchers reconstruct human research artifacts across different stages of the scientific workflow. Using 152 ICLR oral papers as our corpus, we generate 456 research trajectories spanning different stages of the research process. Applying AutoHRS to these trajectories, we find that current Auto Researchers frequently fail to recover key research decisions and their resulting artifacts across multiple stages of the research process.

Mon 9 NovComputation and Language
The gist
Doing scientific research often involves many decisions that humans make, like choosing what problems to study or how to solve them. The paper looks at AI systems called Auto Researchers that try to do research tasks automatically. The authors check how well these AI systems copy the decisions and work that human experts do in AI research. They find that current Auto Researchers often do not make the same choices or produce similar results as humans across different parts of the research process.
Open → 2611.12429v1

Multi agent reasoning predicts breakthrough scientific concept pairs

Bridging Concepts: Multi-Agent Reasoning over Scientific Concept Graphs for High-Impact Research Prediction

Abstract: Within a research community, new topics often emerge when previously unconnected concepts are first studied together. We construct scientific concept graphs (SCGs)--domain knowledge graphs whose nodes are fine-grained research concepts extracted from paper text by LLMs and whose edges capture their co-occurrence in papers--and study the task of forecasting high-impact concept pairs, i.e., discovering concept pairs that are currently unconnected but will have high impact when connected. To this end, we introduce new benchmark datasets constructed from CIKM, KDD, and SIGIR conference papers, with varied prediction settings, including impact forecasting for emerging concepts unseen during training. We propose MiroGraph, a Multi-agent reasoning framework with scientific concept Graphs, in which a feature designer agent designs predictive features for a given concept pair from structural, semantic, and trend perspectives, and then three LLM agents compute the actual feature values from the enclosing subgraph of the concept pair. Using these feature values, a classifier designer agent constructs a classifier that predicts the final impact labels. MiroGraph outperforms various impact forecasting and temporal link prediction methods.

Sun 18 OctArtificial IntelligenceInformation RetrievalMachine Learning
The gist
Sometimes new scientific ideas come from connecting two different concepts that researchers hadn't studied together before. This paper looks at how to predict which pairs of concepts, not yet linked, will become important when combined. The authors build special graphs from research papers to represent these concepts and their connections. They introduce MiroGraph, a system where different intelligent agents create and measure features from these graphs to help predict high-impact connections in science. Their method works better than previous ways of forecasting important new research topics.
Open → 2610.25281v1

Multimodal AI detection improves spotting AI text in math papers

Multimodal AI Detection for Math-Heavy Scientific Text

Abstract: As large language models become increasingly capable of generating, and assisting with, scientific writing, there is a need for greater transparency and disclosure around the use of AI in research. Consequently, a veritable subfield of research and a number of commercial services aim to flag AI-generated content automatically. However, existing detectors which rely only on text extracted from PDFs fail to benefit from the rich visual information contained in the formatting of math-heavy scientific content. To study this, we select human-written proof and prose passages from pre-2022 arXiv LaTeX sources and re-generate them using 15 modern LLMs within the context of the same paper. This leaves us with coupled human and AI outputs in three different representations: the LaTeX source, a rendered image of the PDF, and the text extracted from the PDF. We find that the accuracy of the state-of-the-art Pangram 4 drops significantly as the mathematical content in a passage increases, especially when evaluated on text as opposed to LaTeX. However, this is not an inherent limitation of detecting AI-generated math: finetuning Qwen-3.5-27B based detectors on this domain recovers the gap, with LaTeX inputs performing significantly better than text since they preserve more formatting information. Since LaTeX sources are rarely available in practice, we also train SnapJudge: an image-based detector operating on screenshots of both prose and proofs from papers. At 0.1% false positive rate, SnapJudge achieves a detection rate of 92% on proofs and 93% on prose generated by frontier models, substantially outperforming both Pangram 4 and our text-based detector. We release our data, models and code at https://nlplab.cs.duke.edu/snapjudge.

Thu 15 OctComputation and Language
The gist
As AI systems get better at writing parts of scientific papers, especially those heavy in math, it becomes important to tell AI-generated content from human writing. The authors found that current tools relying only on raw text struggle with math content. They showed that using the original LaTeX code or images of the formatted paper helps detect AI writing much more accurately. They also created a new tool, SnapJudge, which uses screenshots of papers and detects AI-generated math sections better than existing methods.
Open → 2610.21062v1

Scientific taste learned from limited comparisons improves research rankings

Learning Individual Scientific Taste with Privileged Hindsight Distillation

Abstract: Scientific taste shapes which research directions researchers pursue, and it varies across individuals. Existing work evaluates research directions along multiple evaluation criteria or models scientific taste using community feedback signals such as citations and reviews, but neither directly captures how individuals translate assessments under the same criteria into different preferences. We model scientific taste as a researcher-specific mapping from criterion-level assessments in a shared, evidence-grounded evaluation space with proposal-visible evidence, to relative preferences. A population-level mapping learned from retrospective supervision provides a cold start when individual feedback is scarce. We introduce Privileged Hindsight Distillation (PHD), which uses later-emerging scientific evidence during training to improve proposal-time criterion assessment and the population mapping, while exposing only proposal-visible information at inference. We evaluate our approach primarily in quantum research, where structured public reviews are largely unavailable, and use pairwise proposal comparison as a direct interface for observing individual taste. With only 15 comparisons per researcher, our method captures sharply different tastes, including opposing preferences over the same research-direction pairs when they trade off across evaluation criteria. In an interactive study, personalized rankings also surface proposals associated with greater subsequent user engagement, connecting learned taste to downstream research exploration.

Wed 14 OctArtificial IntelligenceMachine Learning
The gist
Different researchers prefer different types of scientific projects, even when judging by the same criteria. The paper shows how to learn these personal preferences by looking at a few examples of what each researcher likes. The method uses extra knowledge available after projects develop, but only uses information available at the time of choosing when making recommendations. Tested mainly on quantum research proposals, it reveals clear individual differences in taste and helps suggest projects that people later engage with more.
Open → 2610.19719v1

LLMs can check research correctness but not its importance

How Well Do LLMs Perform in Peer Review? Evidence from Finance and Accounting

Abstract: Referees increasingly use large language models (LLMs), while finance and accounting journals disagree about whether to allow it. We ask which part of a referee's job the models can do. GPT-4o, whose training data end before the 2024 conference season, referees the papers presented at the 2024 AFA and NBER finance meetings, and we compare its verdicts with publication in the top-three finance journals over the next two years. Papers the model scores one standard deviation higher are 44\% more likely to be published, net of author reputation. The result is not driven by look-ahead bias, as the model does not recall the outcomes. The information comes from its judgements of correctness and exposition. Its judgements of originality and significance carry none, and a referee guide that stresses importance does not help. The split between correctness and contribution holds for accounting papers and for a newer model, GPT-5. LLMs can check whether research is right, but not whether it matters. The technical check can go to the model, while the judgement of contribution stays with people.

Tue 13 OctArtificial Intelligence
The gist
People increasingly use large language models (LLMs) to help review academic papers, but it's unclear what parts of the job these models can do well. The authors tested GPT-4o by having it review finance and accounting papers and then compared its scores to whether those papers were eventually published in top journals. They found the model is good at judging if the research is correct and clearly explained, but not if the research is original or important. This suggests AI can help check technical accuracy, while humans still need to decide what really matters.
Open → 2610.18435v1

Drift Inspector tracks research field changes through contribution claims

Drift Inspector: Exploring and Measuring Scientific Drift with Atomic Contribution Claims

Abstract: Scientific abstracts mix contributions with background, motivation, and meta-language, so tools that read them as-is cannot separate what a field produces from what it discusses. We present Drift Inspector, an open-source system for measuring and exploring how a research field changes over time at the level of Atomic Contribution Claims (ACCs): decontextualized, contribution-bearing propositions an LLM extracts from each abstract before analysis. The system clusters these claims across years into an interactive map where every trend traces back to the claims and papers behind it. Applied to six years of EMNLP, it shows the field shifting away from classic NLP tasks toward LLM-era capabilities such as reasoning and multimodality -- a movement that keyword or whole-abstract counts blur. The released data extend beyond EMNLP: the same pipeline has processed the full ACL Anthology (346k claims, 80k abstracts, 423 venues). Extraction is human-validated and clustering checked against an external manually constructed taxonomy.

Wed 30 SeptComputation and LanguageDigital Libraries
The gist
Scientific papers mix background and contributions, making it hard to see what new ideas a field actually produces over time. The authors created Drift Inspector, a tool that pulls out the core contribution statements from paper abstracts and groups them by year to show how a research area changes. They used it on six years of EMNLP papers to reveal shifts from traditional tasks toward newer capabilities like reasoning and multimodal learning. This approach is clearer than just counting keywords or reading abstracts whole. The tool and data cover a large collection of papers beyond EMNLP.
Open → 2609.39710v1

Consolidated review training improves automated peer review quality

To Consolidate or not to Consolidate? Evaluating the Impact of Consolidation in Multi-Reference Training using Peer Reviews

Abstract: Natural language generation (NLG) tasks span the spectrum of conditional entropy, ranging from highly constrained machine translation to open-ended dialogue generation. Structured tasks like automated peer-review generation occupy the intermediate region, where a single input admits multiple valid, overlapping outputs. In this work, we demonstrate that traditional single- and multi-reference training paradigms are suboptimal for these intermediary tasks. We provide empirical evidence that consolidating diverse references into a unified training signal is crucial for developing effective systems. To facilitate this, we introduce MERC-36K, a large-scale corpus of over 36,000 papers paired with original and consolidated peer reviews. Using this dataset, we train specific architectures to isolate the impact of different reference paradigms and benchmark against existing state-of-the-art systems. Through extensive automatic and human evaluation, we demonstrate that models trained on consolidated references significantly outperform those trained on unconsolidated references. Dataset and code will be released upon acceptance.

Sat 19 SeptComputation and Language
The gist
Generating natural language text can be simple or very open-ended, and some tasks lie in between. The authors studied generating peer reviews for papers, which have multiple ways to give feedback. They found that training computer models using a single combined version of these reviews helps the model do better than using separate, varied reviews. They created a big dataset of papers with original and merged peer reviews to show this clearly. Models trained on these combined reviews produce better outputs according to tests done by both people and automatic measures.
Open → 2609.22805v1

Ai reviewer training risks causing loss of scientific judgment diversity

When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation

Abstract: Large language models (LLMs) increasingly participate in scientific evaluation, both as automated reviewers and as assistants to human reviewers. As model-generated reviews enter public data and future training corpora, AI peer review can become recursive: later reviewers learn from judgments produced by earlier models. We study one step of this feedback loop in a controlled setting. Starting from Llama 3.1 8B, we first fine-tune a reviewer on official ICLR reviews from 2018--2023 and then train four successor models on ICLR 2024 data with systematically varied mixtures of official and model-generated reviews. Our study shows that introducing synthetic reviews compresses rating distributions and reduces both same-paper and corpus-level semantic diversity. We call this pattern $\textbf{scientific-judgment collapse}$. To mitigate this failure mode, we introduce $\textbf{TrustReviewer}$, an open-source LLM-based system for generating peer reviews of AI and machine learning papers. TrustReviewer intervenes at two complementary stages. For training-time prevention, we train the core reviewer in a single stage on a curated corpus designed to reduce low-quality and semantically degenerate supervision. For test-time correction, paired activation steering aims to further mitigate residual tendencies toward collapsed judgments without further training or additional expert annotation. Together, these results characterize a concrete risk of recursive reviewer training and provide practical interventions for preserving judgment diversity and improving recommendation alignment in AI-assisted scientific evaluation.

Thu 17 SeptMachine Learning
The gist
AI models are starting to review scientific papers, but when these AI-generated reviews are used to train new AI reviewers, the variety and richness of their judgments shrink. The authors call this problem scientific-judgment collapse. They studied this effect using several versions of AI models trained on real and synthetic reviews. To fix it, they created a system called TrustReviewer that carefully curates training data and adjusts AI behavior during review to keep diversity and quality high.
Open → 2609.20942v1

GreCAPTCHA tests authors understanding to verify research ownership

greCAPTCHA: Assessing Understanding as Evidence of Research Authorship Under Generative AI

Abstract: Conferences, journals, funders, schools, and universities are struggling with a surge of potentially AI-generated submissions from ostensibly human authors, who may not have exercised sufficient human oversight for their manuscripts. In turn, institutions evaluating submissions can no longer reliably credit expertise based solely on authors' names on submitted work. To address this problem, we propose greCAPTCHA, a proctored assessment approach that measures authors' understanding of research manuscripts via the construct of capacity to verify, which we define as the knowledge and reasoning required to critically assess the contents underlying one's contributions to a manuscript. greCAPTCHA generates questions assessing multiple levels of understanding and provides an evaluative report based on authors' responses. Using a prototype implementation, we conduct a user study and semi-structured interviews with $31$ researchers to evaluate greCAPTCHA. Its automated scores predict which papers were or were not authored by study participants with an AUC of $0.90$. Participants reported positive overall experiences with the system and remarked on the appropriate construct validity for author understanding, while also suggesting important changes to be made before deployment. Our results provide initial evidence that greCAPTCHA can assess manuscript-specific understanding under proctored conditions.

Thu 17 SeptDigital LibrariesArtificial IntelligenceHuman-Computer Interaction
The gist
It can be hard to tell if a research paper was really written by the person named as the author, especially with AI tools helping to write text. The authors created greCAPTCHA, a test that asks questions about a research paper to see if the author really understands it well. When tried out with 31 researchers, the test could accurately tell which papers they wrote based on their answers. People who used it said the questions made sense and suggested improvements. This tool can help universities and journals check if authors truly know their own work.
Open → 2609.20481v1