Papers for

journal editorial teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Ai models vary widely in spotting errors in scientific paper reviews

Auditing AI peer reviewers: a benchmark on real scientific papers

Abstract: LLM-based peer review is now used at production scale, but its evaluation remains limited. We introduce a benchmark that injects objective, localisable errors of four types (general issues, mathematics, numerics, citations) at three severity levels into real papers and measures each reviewer's detection rate by error type and severity, together with its false-positive rate on clean copies. We also present Skepthical, a multi-agent reviewer that verifies citation-backed claims by retrieving the cited papers, mathematics with computer algebra, and numerical claims with executed code. We apply the benchmark to two corpora of 15 recent arXiv preprints each, one in astrophysics and one in machine learning (ML), injecting a total of 357 errors, and evaluate twelve reviewer systems: six single-LLM reviewers, two Claude Code configurations, three publicly available reviewers, and Skepthical. The choice of model has the largest impact on detection rate: given the same prompt, three frontier LLMs differ by about 50 percentage points in total detection on both corpora. Tools and scaffolding help when they contribute what the model cannot supply on its own. In particular, on astrophysics, Claude Code improves on a single Claude call in mathematics and numerics detection, and enforced citation retrieval gives Skepthical the highest citation error detection rate of all systems by a large margin. This citation lead, however, is not observed on ML, where the models appear to know the literature better and Skepthical's citation detection rate is statistically indistinguishable from that of the best single LLM. False-positive rates are below 10% for most systems, but vary with prompt, model, and scaffolding, with mathematics flags the main source of false positives. The evaluation pipeline, the injected errors, and the adjudication logs are released as a benchmark.

Thu 12 NovComputation and LanguageArtificial IntelligenceDigital Libraries
The gist
Checking scientific papers carefully for mistakes is a big job that some AI tools now help with. The authors created a test with real scientific papers that had made-up errors added, to see how well different AI reviewers could find those mistakes. They found that the choice of AI model makes a big difference—some models find many errors while others miss half. They also made a special AI reviewer called Skepthical that checks claims by looking up citations and verifying math and numbers, which worked especially well for spotting citation problems in physics papers. Most AI reviewers rarely say there’s a mistake when there isn’t one, but math errors are the trickiest for them.
Open → 2611.17052v1

Peer review comments split into claims checks and value judgments

ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments

Abstract: Peer-review comments check what a paper does and judge why it matters, but no existing scheme distinguishes these two evaluative acts. We argue that making these acts explicit and measuring how reliably LLMs identify them is a prerequisite for responsible LLM-assisted review analysis: humans need to inspect review judgments and their grounds. This defines a more honest mission for NLP in peer review than full automation. We present ReviewMap, a framework that classifies each comment as assessment of claims and scholarly standards or calibration of contribution value and venue fit, with finer-grained attributes applied conditionally by act type. Applied to 500 ICLR~2026 papers through multi-model LLM consensus, the act layer proves reliably automatable under the operational guideline (inter-model $κ\geq 0.75$), supported by five-human validation on 470 development comments, while the attribute layer requires human oversight, defining a measured automation boundary. At the corpus level, 17% of reviewer-paper instances contain no panel-coded calibration, and the panel-coded calibration share is nearly $3\times$ as high in Strengths as in Weaknesses. This boundary informs three affordances: a review guideline that operationalizes the assessment-then-calibration workflow for reviewers, a per-comment rebuttal strategy scaffold for authors, and a discussion-phase dashboard for area chairs.

Thu 5 NovComputation and LanguageHuman-Computer Interaction
The gist
Peer reviews of scientific papers often mix two different kinds of comments: checking what the paper claims and judging how valuable or suitable it is. The authors created ReviewMap, a system that sorts review comments into these two types to make it clearer. They found that large language models (LLMs) can reliably do the main sorting, but humans still need to check the details. This approach helps make review feedback easier to understand and manage for everyone involved.
Open → 2611.07489v1

AI reviewers may not notice major improvements in revised papers

Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories

Abstract: AI reviewers are increasingly used to assess scientific manuscripts, yet it remains unclear whether their judgments reflect the improvements authors make through revision. We investigate this question using PaperTrajectory-2026, the 2026 release of PaperTrajectory, a dataset of 6,959 paper trajectories across five AI conferences, with automatic annotations of changes along 14 dimensions and separate judgments of overall quality. The annotations show that major improvements emphasize empirical evidence and framing, while later revisions tend toward incremental improvement. Using independent assessments of manuscript versions from eight AI reviewers, we examine whether annotated major improvements receive higher scores, which revision dimensions are associated with score changes, and whether user opinions can outweigh observed revision gains. Every reviewer leaves scores unchanged on most major improvements, and agreement on the direction of score changes is weak. Revision dimensions show some reviewer-specific associations with score increases, but provide limited predictive discrimination and no consistent guide across reviewers. Even when a revision earns a higher neutral score, a brief negative user opinion can erase or reverse that advantage, with substantial variation across reviewers. These findings show that substantive revisions are not consistently reflected in review scores and that observed scoring gains can be vulnerable to user framing, motivating assessments that connect manuscript evidence to overall judgments and remain stable when no scientific evidence changes.

Sun 1 NovComputation and Language
The gist
AI systems are used to review scientific papers, but it is unclear if they recognize when authors make big improvements after revising. The authors studied thousands of papers and their revisions, comparing these to scores given by AI reviewers. They found that AI scores often do not change much even when big improvements are made, and reviewers sometimes disagree on how scores should change. Also, a brief negative comment from a user can outweigh any positive change in scoring. This suggests that AI reviews may not consistently reflect how much a paper improves.
Open → 2611.02244v1

Peer review funnel redesign drastically cuts workload and improves top paper selection

Change the Funnel, Not Who May Submit: Engineering Peer Review for the Flood

Abstract: Every reviewer knows the feeling: more papers each cycle, three full reviews for each of them, and a suspicion that near the accept line the outcome is luck. The numbers bear it out. NeurIPS received 30,709 submissions in 2026, more than three times as many as in 2021, and reviewing them took some 184,000 expert hours, or 92 years of researchers' time. Three-quarters of those hours went to papers that were rejected. And when NeurIPS twice had the same papers reviewed by two committees, the second rejected half of what the first accepted. Conferences are responding by limiting who may submit. We simulated that answer, and an alternative, in a model calibrated to the NeurIPS experiments. The limits save little and select no better, and those based on a past acceptance fall mostly on new authors. The alternative changes how papers are judged, spending full reads only where a decision turns on them. In a staged funnel, an automated quality check first turns back papers that are not ready. The rest are ranked by their value, from short summaries, by readers that each submission brings. Only the finalists get a full read, which checks their claims and settles the final ranking. The staged funnel needs a little over half of today's expert hours, accepts 95.5% of papers in the top 3-10% (against 74% today), and makes nearly every top-0.5% paper a spotlight or oral. And because every paper brings its own reader, the funnel keeps pace with the flood.

Sat 31 OctMachine LearningComputers and SocietyDigital Libraries
The gist
As more papers flood into big conferences like NeurIPS, reviewers spend many years worth of time reading most papers that get rejected. The authors show that simply limiting who can submit does not help much and unfairly hurts new authors. Instead, they propose a multi-stage review process where an automated check filters out the weakest papers first, then many readers rank the rest from short summaries, and only the finalists get full reviews. This approach reduces required expert hours by about half while accepting more top-quality papers and highlighting the very best consistently.
Open → 2611.01076v1

Llm evaluation methods show limits in predicting paper quality and impact

Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment

Abstract: Growing submission volumes have renewed interest in using large language models (LLMs) to support peer review. It remains unclear how such evaluation should be structured, how much the model matters, and whether LLM scores predict research impact. We evaluate five paradigms across three input representations and three LLMs on 232 stratified ICLR 2025 submissions. Direct absolute rating matches comparative ranking on rank correlation with human scores, yet compresses most predictions above the accept threshold, flagging almost none of the papers reviewers scored below it. A base-rate-matched recalibration control locates the cause: comparative ranking supplies placement, not better ordering, and after recalibration Direct Rating reaches higher agreement than Elo while making 24% as many API calls. Abstract-only input beats full-paper summarization, and model choice matters as much as paradigm choice. Replicating on all 11,513 scored submissions reproduces these patterns, shows absolute agreement depends on the evaluated set's score spread, and finds none beating an always-reject baseline in accuracy. Human scores track citations better than any paradigm, with one exception: Group Rating retains its citation correlation after controlling for human scores, and alone ranks within the accepted pool. Ranking quality alone cannot assess review quality; calibration, thresholds, model, and target outcome all matter.

Fri 30 OctComputation and LanguageDigital Libraries
The gist
Figuring out which academic papers are good is hard and important for researchers. This study looks at different ways to use large language models (LLMs) to rate academic papers and sees how well these ratings match human reviewers' opinions and future impact like citations. The authors find that simple direct ratings by LLMs often cluster scores above acceptance thresholds and miss weaker papers, while ranking methods don't improve ordering much. The choice of LLM and how the paper is represented matter a lot, and none of the methods beat a simple rule of always rejecting papers in accuracy. Human scores still predict citations better than these LLM methods.
Open → 2610.41391v1

Large language models give harsh and inconsistent reviews of top science papers

Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals

Abstract: With the growth of scientific output, the peer review system is approaching a breaking point: reviewers are increasingly hard to secure, and turnaround times lengthen accordingly. LLM-assisted or LLM-generated peer review has been proposed as a remedy, but how such models would treat manuscripts already published in top-tier journals, the closest thing the current system has to a gold standard of scientific validity, is not known. Here we assembled a corpus of 200 articles, each released under a CC BY 4.0 license and published in 2026, from 20 general-science, life-science, and clinical-medicine journals including Nature, Science, and Cell, and asked four current LLMs (two proprietary, two open-weight) to produce independent peer-review reports of each one under a blinded protocol. Their decisions were overwhelmingly harsh and starkly inconsistent with each other: 605 of 800 reviews (75.6%) recommended major revision or rejection, but the rate at which any single model did so ranged from 17.5% to 100% (p < 0.001), disagreement on manuscripts that had, by construction, already survived real editorial review at the journals in question. When the same manuscripts were doctored with one deliberately planted, field-specific methodological or statistical flaw, detection rates varied from 17.0% to 60.5% across models (p < 0.001). Recommending revision for papers that had already passed editorial review (low specificity) while missing flaws deliberately placed in front of them (low sensitivity), these models produce output that should be read for what it is: an uneven, framing- and model-dependent signal.

Thu 22 OctDigital LibrariesArtificial Intelligence
The gist
Scientific papers are usually checked by human experts before they get published to make sure the science is good. This paper looked at how well big language models (LLMs), which can read and write text like humans, do when asked to review already published top science papers. The LLMs often suggested big changes or rejection, even though the papers had already passed expert review. They also often missed important errors that were purposely added to test them. This means LLMs currently give mixed and unreliable reviews if used to judge published science.
Open → 2610.31414v1

Premisecheck speeds fact checking of human and ai peer reviews

PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

Abstract: Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.

Thu 15 OctComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Review comments on scientific papers sometimes mistakenly say things the paper doesn’t actually say. The authors introduce PremiseCheck, a method that breaks down these review comments into clear statements and checks each one against the paper using a fast decision model. This approach is almost as accurate as a large language model that reads the whole paper but is cheaper and faster. It works well across fields like natural language processing and natural sciences. This helps make peer review more reliable and efficient.
Open → 2610.21046v1

LLMs can check research correctness but not its importance

How Well Do LLMs Perform in Peer Review? Evidence from Finance and Accounting

Abstract: Referees increasingly use large language models (LLMs), while finance and accounting journals disagree about whether to allow it. We ask which part of a referee's job the models can do. GPT-4o, whose training data end before the 2024 conference season, referees the papers presented at the 2024 AFA and NBER finance meetings, and we compare its verdicts with publication in the top-three finance journals over the next two years. Papers the model scores one standard deviation higher are 44\% more likely to be published, net of author reputation. The result is not driven by look-ahead bias, as the model does not recall the outcomes. The information comes from its judgements of correctness and exposition. Its judgements of originality and significance carry none, and a referee guide that stresses importance does not help. The split between correctness and contribution holds for accounting papers and for a newer model, GPT-5. LLMs can check whether research is right, but not whether it matters. The technical check can go to the model, while the judgement of contribution stays with people.

Tue 13 OctArtificial Intelligence
The gist
People increasingly use large language models (LLMs) to help review academic papers, but it's unclear what parts of the job these models can do well. The authors tested GPT-4o by having it review finance and accounting papers and then compared its scores to whether those papers were eventually published in top journals. They found the model is good at judging if the research is correct and clearly explained, but not if the research is original or important. This suggests AI can help check technical accuracy, while humans still need to decide what really matters.
Open → 2610.18435v1

LLM framework improves error detection in scientific peer review

Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review

Abstract: The rapid growth of scientific publishing has strained peer review, particularly in machine learning, raising concerns about declining review quality and increasing reviewer workload. Large language models (LLMs) have been proposed as automated review assistants, yet their evaluation has focused largely on imitating human-written reviews rather than supporting the core functions of peer review. Here, we introduce a verification-centric perspective on LLM-assisted peer review, emphasizing error detection as a critical and resource-intensive task. We present a scalable benchmark that evaluates review systems' ability to identify logical contradictions, constructed through synthetic insertion of errors into conference papers, yielding unambiguous evaluation targets and enabling systematic comparison. We further propose a Multi-Layered Review (MLR) framework that prioritizes detailed manuscript comprehension before review generation, aligning more closely with human reviewing practices while improving token efficiency. Across evaluations, our approach demonstrates strong alignment with human review scores, achieves high error detection performance, and provides complementary perspectives on reviewer focus. These improvements can be attributed to both the choice of the underlying LLM and the design of our system. At the same time, we corroborate persistent vulnerabilities to adversarial manipulation, underscoring the need for robustness in automated review systems. Our findings highlight the importance of rigorous, error-focused evaluation to guide responsible deployment of LLM-based tools in peer review and other critical scientific workflows.

Thu 8 OctArtificial Intelligence
The gist
As more scientific papers are published, checking their quality through peer review becomes harder and slower. The authors created a new way to help AI language models do peer reviews better by focusing on finding mistakes in papers, not just copying human reviews. They built tests by purposely adding errors to papers and developed a step-by-step review process that understands the paper well before giving feedback. This method helps AI give reviews that agree more with human reviewers and catch errors more accurately, though some weaknesses remain that need fixing.
Open → 2610.11087v1

Peer review noise causes 23 to 30 percent disagreement on paper acceptance

The Review Lottery: Calibrating an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

Abstract: How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of accepted papers would be rejected. External calibration: at the NeurIPS 2021 reviewer-count caliber (k=3), the simulated 2021 disagreement rate is 23.3% [21.7%, 25.0%] vs. reported 23.0% (bias +0.3pp); accept precision and committee correlation agree within 5pp and 0.04. Internal calibration: on 18,740 papers with 4+ reviews, random model-free 2+2 reviewer splits agree with the k=2 simulation within 1pp in 2018 and 2021-2025. Longitudinally, we find no robust time trend in reviewer noise over 2017-2025. The high accepted-paper flip rates of 2020 and 2021 have distinct mechanisms: the 2020 four-point scale compressed scores (23.7% of papers had zero within-paper variance), and a counterfactual shows coarsening the scale raises disagreement by about 7pp; 2021 instead combined the lowest signal-to-noise ratio in the sample with the most threshold-crowded acceptances. For the LLM era, a 2023 breakpoint test on within-paper score variance finds no break, but the design has almost no power, and no post-2022 review text or confidence data exist, so no LLM attribution is attempted.

Mon 5 OctArtificial Intelligence
The gist
It is hard to tell how much different sets of reviewers might disagree on accepting or rejecting the same research paper because running two independent review committees is very expensive. The authors developed a new way to estimate this disagreement using only public review data from the ICLR conference over nine years. They found that about 23-30% of decisions would differ if a paper was reviewed by a different group, meaning many accepted papers might be rejected by others. Their method was tested by comparing with actual data where available and showed no clear change in disagreement over time.
Open → 2610.06591v1

Policy conditioned AI use detection improves academic publishing compliance

Policy-Conditioned AI-Use Detection: An Evidentiary Framework for Academic Publishing

Abstract: Major venues now publish detailed rules about how authors, reviewers, and area chairs may use AI, and those rules differ by role, by task, and by what must be disclosed. AI detection, the instrument usually proposed to enforce them, estimates something else: whether an AI model wrote the text. We argue that this target is misaligned with the decisions conferences and journals face, and propose policy-conditioned AI-use detection, an evidentiary framework for assessing whether a human--AI workflow complied with a stated rule. Policy makes the governing rule an explicit input. Inference reports hypotheses, evidence, calibration regime, and uncertainty in place of verdicts such as "AI detected". Evaluation builds benchmarks from reproducible pipelines that generate compliant and non-compliant workflows, and reports true positive rate at a false positive rate the venue fixes in advance. We work the framework through peer review, where at plausible violation rates a detector at a strong operating point still flags more compliant authors than violating ones. The framework therefore also names what a venue must instrument: structured disclosure, approved-tool routing that respects reviewer confidentiality, and a path by which a finding can be contested. Under this framing a detector is not an authorship classifier but an auditable procedure with an error rate the venue fixes in advance and can defend.

Tue 29 SeptComputation and LanguageArtificial IntelligenceComputers and Society
The gist
Academic conferences and journals now have detailed rules about how people can use AI tools, but current AI detectors only say if AI likely wrote a text, which is not enough. The authors suggest a new way to detect AI use that checks if specific rules were followed, considering different roles and tasks. Instead of just labeling text as AI-generated, this method reports how confident it is and what evidence it used. They tested this approach in peer review settings and found it can mistakenly flag more authors who follow the rules than those who break them, showing that policies and other steps must support the detection.
Open → 2609.38427v1

System improves meta-review writing by handling conflicting peer reviews

When Evidence Conflicts: Reliability-aware Meta-review Generation

Abstract: Generating coherent meta-reviews from multiple peer reviews is challenging when reviewer evidence conflicts and varies in reliability. Existing approaches typically formulate meta-review generation as a multi-document summarization task and aggregate reviewer feedback uniformly, making it difficult to determine which opinions should be prioritized under disagreement. In this paper, we study meta-review generation through reliability-aware evidence aggregation. Our framework first extracts aspect-level opinions from peer reviews and identifies conflicting evidence within each aspect. It then estimates opinion-level support and review-level quality to measure evidence reliability. Based on these signals, the framework assigns reliability-aware weights to reviewer feedback, enabling the generator to prioritize better-supported arguments while preserving diverse perspectives. Experiments demonstrate that our method consistently improves meta-review generation over strong baselines on both automatic and human evaluations, with clear gains in conflict recognition and resolution under high-conflict review scenarios. The code and implementation details are publicly available at https://github.com/Wangxz729/reliability-aware-meta-review.

Mon 21 SeptComputation and Language
The gist
Writing a summary from multiple reviews is tricky when the opinions disagree. The authors created a system that checks how reliable each review is and picks the most supported points to include. This way, the summary better balances different views while highlighting strong evidence. Their tests show this works better than older methods, especially when reviews conflict a lot.
Open → 2609.24028v1

GreCAPTCHA tests authors understanding to verify research ownership

greCAPTCHA: Assessing Understanding as Evidence of Research Authorship Under Generative AI

Abstract: Conferences, journals, funders, schools, and universities are struggling with a surge of potentially AI-generated submissions from ostensibly human authors, who may not have exercised sufficient human oversight for their manuscripts. In turn, institutions evaluating submissions can no longer reliably credit expertise based solely on authors' names on submitted work. To address this problem, we propose greCAPTCHA, a proctored assessment approach that measures authors' understanding of research manuscripts via the construct of capacity to verify, which we define as the knowledge and reasoning required to critically assess the contents underlying one's contributions to a manuscript. greCAPTCHA generates questions assessing multiple levels of understanding and provides an evaluative report based on authors' responses. Using a prototype implementation, we conduct a user study and semi-structured interviews with $31$ researchers to evaluate greCAPTCHA. Its automated scores predict which papers were or were not authored by study participants with an AUC of $0.90$. Participants reported positive overall experiences with the system and remarked on the appropriate construct validity for author understanding, while also suggesting important changes to be made before deployment. Our results provide initial evidence that greCAPTCHA can assess manuscript-specific understanding under proctored conditions.

Thu 17 SeptDigital LibrariesArtificial IntelligenceHuman-Computer Interaction
The gist
It can be hard to tell if a research paper was really written by the person named as the author, especially with AI tools helping to write text. The authors created greCAPTCHA, a test that asks questions about a research paper to see if the author really understands it well. When tried out with 31 researchers, the test could accurately tell which papers they wrote based on their answers. People who used it said the questions made sense and suggested improvements. This tool can help universities and journals check if authors truly know their own work.
Open → 2609.20481v1

Refverifier aids peer reviewers in checking scientific citations accurately

RefVerifier: Semi-Automated Reference Claim Verification for Scientific Manuscripts

Abstract: As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing.

Mon 7 SeptSoftware Engineering
The gist
Scientific papers often include claims supported by citations, but verifying these citations takes a lot of time, which reviewers may not have. The authors created RefVerifier, a tool that helps reviewers check if a paper’s claims match the cited sources by automatically finding and summarizing the relevant parts of cited papers. RefVerifier can find citation information, retrieve related open-access papers, and highlight evidence passages, helping reviewers make faster and more accurate judgments on citation reliability. This approach does not replace reviewers but assists them by automating some tedious steps.
Open → 2609.07652v1