Papers for

legal document reviewers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Ai model learns to judge paper soundness beyond human reviews

PaperCritic: Training AI Reviewers to Assess the Soundness of Papers

Abstract: As AI conference submissions grow rapidly, AI reviewers are being considered as an alternative. However, simply imitating human reviewing is problematic because reviewing serves multiple purposes, including evaluating papers' soundness, interestingness, and clarity, the last two of which are subjective. In this work, we focus solely on evaluating soundness, the most important and objective criterion, and define the learning objective as jointly maximizing (1) precision, based on the soundness and substantiveness of review points, and (2) recall, the coverage of high-quality reference review points. Under this problem formulation, we construct a quality-labeled review point dataset, PaperSound-23K, using 1K ICLR submissions and 23K human-written and LLM-generated review points. Using this dataset, we train an AI reviewer model, PaperCritic, through reinforcement learning with the F1 score of precision and recall as the reward. Our experiments show that PaperCritic outperforms frontier models and models trained with other AI reviewer training methods on an in-domain test set. Surprisingly, learning to review ICLR submissions also transfers to reviewing Nature-family papers and even to out-of-domain reviewing tasks (e.g., code review, legal contract review, and feedback on story writing).

Fri 13 NovComputation and Language
The gist
Checking scientific papers involves many different feelings, but one of the most important parts is deciding if the research is done properly, which is more objective. The authors focused on training an AI to spot this 'soundness' by teaching it with many real and AI-made review comments labeled for quality. They created a big dataset from past conference papers and used this to train their AI reviewer called PaperCritic. This AI not only got better at checking conference papers but also surprisingly worked well on other types of documents like legal contracts and stories.
Open → 2611.17834v1

Multi-agent screening shows limits of agreement and acceptance evidence

Multi-Agent Screening: What Agreement and Acceptance Cannot Tell You

Abstract: When AI agents screen items for a costly human assessor, as in agentic search, e-discovery and AI reviewing of papers, only what they pass gets judged. We show what each kind of evidence can establish about their errors, even with unlimited data. Agreement shows that agents concur, not that they are right, however many there are. Acceptance by the assessor gives precision but only a lower bound on recall. Knowing which agents found which relevant items gives an upper bound on recall if agents' finds are not negatively correlated, and caps what one more agent can add if agents are interchangeable. Without assumptions about the agents, recall can be pinned down only by judging items they did not pass: a random sample of these estimates how many relevant items were missed, to any chosen accuracy, and the sample needed does not grow with the number of agents.

Mon 9 NovMultiagent SystemsArtificial IntelligenceInformation Retrieval
The gist
When multiple AI agents are used to filter items for a human reviewer—like sorting documents or papers—only the accepted items get reviewed. The authors show that just because agents agree on an item, it doesn't mean they are correct. Also, seeing which items the human accepts tells us about accuracy but only partly about how many relevant items are found. To estimate how many relevant items might be missed, it’s necessary to check a random sample of the items rejected by the agents. This method works regardless of the number of agents involved.
Open → 2611.12705v1

Detecting AI text struggles with very short passages

How Short Is Too Short? Length-Conditional Calibration and Abstention for Detecting AI-Generated Text

Abstract: Detectors of AI-generated text are usually tuned and tested on passages of several hundred tokens, yet many real cases are short. We study score-based detection as a function of passage length. We give a closed form for how the false-positive rate of a threshold set on long text changes as length falls, and a formula for the number of tokens a detector needs to reach a target true-positive rate. We calibrate thresholds within length bins with conformal p-values, and we add a bin-level rule that abstains when the detector cannot be informative at that length. On synthetic scores the formulas agree with simulation, and binned calibration holds the false-positive rate at its nominal value at every length. On three zero-shot detectors (log-likelihood, log-rank, Binoculars), three corpora and two open-weight generators, binned calibration gives a mean false-positive rate of 5.0% in every detector-length cell from 10 to 200 tokens, averaged over 300 random document-level resplits. A single threshold set at 200 tokens gives 29% for Binoculars at 10 tokens and about 1% for log-likelihood, as the closed form predicts. No detector reaches an 80% true-positive rate, and the best, Binoculars at 200 tokens, reaches 59%, so the rule abstains in every cell at that floor and releases only the 200-token bins at floors of 0.3 and 0.5.

Sat 7 NovArtificial Intelligence
The gist
Detecting whether text is written by AI works well when the text is long, but it’s much harder when the passages are short. The paper shows formulas to predict how detection errors change as text gets shorter. The authors create a method that adjusts detection settings depending on passage length and can say 'I don't know' when the text is too short to decide. Tests show their approach keeps error rates steady across different lengths but that AI text detection is limited for very short pieces.
Open → 2611.10901v1

Semantic conditional similarity distinguishes document contributions better

Semantic Conditional Similarity (SCS): A Formal Mathematical Framework

Abstract: Semantic Textual Similarity (STS) operates on document pairs, returning a single scalar per comparison. In a multi-document setting where several sources may have contributed to one or more targets, pairwise STS cannot disentangle each source's marginal semantic contribution from their shared vocabulary. In this paper, Semantic Conditional Similarity (SCS) is introduced as a conditional form of STS that computes similarity between two documents after eliminating semantic elements shared with a third via set-theoretic operations. SCS makes four contributions: 1) it introduces directionality, partitioning pairwise relations into lateral (inter-source) and vertical (source-to-target) classes; 2) conditioning induces role-dependent asymmetry by extending the comparison domain from unordered pairs to ordered, conditioned triples of a target, a reference, and a conditioning document; 3) SCS introduces inter-document conditioning by subtracting shared content from a third document, complementing the intra-document conditioning in recent C-STS work; and 4) the set-theoretic decomposition achieves threshold-free partitioning of shared and residual content, requiring no arbitrary similarity cutoff. Although developed for the $n \to 1$ configuration, the framework extends to $1 \to n$ via vertical role reversal and to $m \to n$ via independent composition of source- and target-side conditioning. With the established $1 \to 1$ case, these span the full spectrum of document cardinality. SCS scores should be interpreted ordinally, as rankings of marginal contributions rather than absolute measurements. The broader implication is that metric space axioms commonly assumed in similarity formalizations constrain expressiveness in role-differentiated multi-document settings, and that extending the domain to conditioned triples can recover information that pairwise-symmetric comparisons suppress.

Thu 5 NovComputation and Language
The gist
Measuring how similar two texts are usually gives just one overall score, but that doesn't explain what parts came from which sources when many documents are involved. The authors propose Semantic Conditional Similarity, a new method that compares two documents while removing the shared content from a third, helping to identify each source’s unique contribution. This approach adds direction to comparisons and avoids arbitrary thresholds, giving a clearer ranking of how each source impacts a target document. It works for different numbers of source and target documents and reveals information that simple pairwise comparisons miss.
Open → 2611.08778v1

Ai humanizers often change factual details in rewritten text

Humanized but Still True? Auditing Facts and Detection in AI-Text Humanizers

Abstract: AI "humanizers" rewrite machine-generated text so that detectors classify it as human-written. Public leaderboards score evasion and embedding similarity, but not whether checkable details survive. We check whether numbers, dates, person and organization names, and quoted spans survive each rewrite, using a rule-based checker validated by two AI annotators (precision 0.84-1.00, recall 0.93-0.95; no human annotation). In the October 2026 cycle of a public humanizer leaderboard, all 11 commercial tools changed or dropped at least one of these units in 35-65% of the documents that contained one. Most changes were to quoted spans; counting numbers and names alone, 10 of the 11 tools still altered at least one in 20% or more of the documents. The September 2026 cycle was similar (19-50%). An open 12B humanizer kept at least 97% of numbers and names in fact-dense U.S. federal agency text, but reworded 67-78% of the quoted spans, against its own prompt. Unit retention and detector evasion were not clearly related across tools (Spearman rho = -0.46, 95% CI -0.65 to 0.08), and the open model's four released files (two of them quantization-aware low-bit files) did not show the pre-registered monotone trade-off. On federal texts with matched human, AI-styled and humanized versions, a general-purpose decision model, Jev, asked a single zero-shot question, separated human originals from humanized rewrites with AUROC 0.864 (0.827-0.902), 0.18-0.22 higher than Binoculars, Fast-DetectGPT and the adversarially trained RADAR, while calling 24% of the human originals AI at a 0.5 threshold. On the AI-styled rewrites by 2026 models, both perplexity-based detectors scored below chance (AUROC 0.37 and 0.39), also with line breaks removed. Hypotheses, data rules and analyses were frozen before the main runs; two earlier data contacts are disclosed. We release the checker and the per-item results.

Fri 30 OctComputation and LanguageCryptography and Security
The gist
AI tools that try to make machine-written text look human often change important facts like numbers, dates, or names. The authors tested 11 commercial tools and found many changed facts in 20% to 65% of cases, especially things that were quoted exactly. They also showed that current AI detectors struggle to tell humanized text from human-written text, but a new method called Jev worked better. The authors provide a tool to check if facts survive rewriting.
Open → 2611.00359v1

When large language model answers vary and may be unreliable

When the Same Prompt Gives Different Answers (and the Same Answer Isn't Always Right): A Taxonomy of Failure Mechanisms, Testing, and Mitigation Strategies for LLM-Assisted Evidence Synthesis Read-and-Evaluate Tasks

Abstract: LLM read-and-evaluate tasks include study screening, data extraction, risk-of-bias assessment, and critical appraisal in the evidence synthesis field. These task differ in the amount of LLM interpretation they require. Some tasks involve retrieving relatively simple information, such as an author name, sample size, or publication year. Others require the LLM to read source material, interpret it against predefined criteria, and make a judgement. Researchers should assess whether LLM judgements are accurate, supported by appropriate source evidence, and reproducible under specified conditions. A correct judgement does not necessarily indicate that the underlying evaluation process is reliable. We developed a taxonomy for identifying mechanisms that may compromise these properties and for informing the selection and reporting of appropriate evaluation methods. The taxonomy outlines potential ways that LLM judgments from read-and-evaluate tasks can fail, called TRAILS (Taxonomy of Reliability And Instability in LLM-assisted evidence Synthesis). The TRAILS taxonomy addresses two complementary questions. First, what mechanisms can cause the same read-and-evaluate task to produce different judgements across repeated LLM runs? Second, what mechanisms can systematically affect the judgement or required output in a way that is not apparent from the judgement itself? This study presents 38 mechanisms and identifies the deployment settings in which each can operate, approaches for testing them, and mitigation strategies. We also show why repeatability or accuracy does not rule out a hidden judgement failure and why human evaluation should assess whether a judgement is grounded in the supplied evidence, independent of inappropriate external influence, and produced through a process that can be examined and verified.

Thu 29 OctArtificial IntelligencePerformance
The gist
Large language models (LLMs) are increasingly used to help read and evaluate scientific studies, but these models can sometimes give different answers when asked the same question more than once. The authors developed a classification system called TRAILS that explains why these variations happen and how errors can occur even when answers seem consistent or correct. They highlight the importance of checking that LLM answers are supported by evidence and follow a reliable reasoning process. The work also suggests ways to test LLM outputs and reduce mistakes in evidence synthesis tasks.
Open → 2610.39070v1

FilingLoop improves review of corporate documents with evolving AI agents

FilingLoop: Evaluating and Evolving LLM Agents for Corporate Disclosure Review and Revision

Abstract: Public disclosure repositories such as SEC EDGAR support informed investment through web access to corporate filings. Supporting their review with language model agents requires discovering unstated deficiencies across related documents, acquiring missing company facts, and verifying that requested revisions satisfy evidence-backed requirements. Existing financial question-answering evaluations provide limited evidence of these interactive capabilities. We introduce \textbf{FilingLoop}, an interactive benchmark and a self-evolving agent framework for corporate disclosure review and revision. Grounded in historical U.S.\ Securities and Exchange Commission correspondence, FilingLoop reconstructs 380 predominantly IPO review tasks from 96 companies under distinct information constraints. An evaluator separate from the acting agents scores final filings and responses against 6,099 eligible requirements and audits unsupported claims and harmful revisions. Our framework couples iterative review and revision within each case with procedural learning across cases, converting requirement-level evaluation feedback into role-specific guidance without updating model weights. Updates affect only subsequent cases, allowing procedural adaptation to be assessed without retrying scored tasks. On the evaluated stream, evolution increases requirement fulfillment from 24.28\% to 27.96\% and full comment resolution from 16.90\% to 20.75\%, with a 5.8\% increase in total cost over the fixed-policy variant. FilingLoop connects the concerns an agent discovers with the company facts it obtains and the revisions it commits, providing requirement-level feedback for procedures used in later review tasks.

Tue 27 OctArtificial Intelligence
The gist
Corporate documents filed with the U.S. Securities and Exchange Commission often need careful review and revision. To help with this, the authors developed FilingLoop, a system that uses AI agents to check filings for missing information or errors and suggest fixes. The system learns from past reviews without changing its underlying AI model, gradually improving how well it meets regulatory requirements. The authors showed that over time, FilingLoop was able to resolve more review comments and fulfill more document requirements.
Open → 2610.37905v1

Multi agent large language models increase defects found but also false alarms

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task

Abstract: Practitioner guidance holds that a large language model reviews a document better when the work is divided among agents. We test a narrow form of it in a controlled comparison of single-agent and multi-agent setups on a financial task, reviewing credit assessment memos against requirements. One agent checks six requirements in one call. A team of three on the same model checks the same memo, two requirements each, in the same words, with no roles or later stage. Labels are known for every requirement on 300 synthetic memos, so misses and false alarms are measured separately. Two primary predictions are tested on four models at a Bonferroni-corrected level. Under an instruction to decide each requirement once, the team catches 0.08 to 0.10 more of the defects that carry an explanation on the three models whose single agent misses some, at the corrected level on two. It raises more false alarms on all four, at the corrected level on three. Balanced accuracy, an exploratory measure, changes by 0.02 or less, and the team uses 2.2 to 2.4 times the tokens. Showing each agent the requirements it was not given reduces the added false alarms on the two Claude models, at the corrected level on one. In exploratory comparisons, sampling the single agent three times and flagging on any draw gives higher balanced accuracy than the team on those models, and the team's results depend on the instruction.

Tue 20 OctArtificial IntelligenceComputational Engineering, Finance, and Science
The gist
This paper studies whether splitting a large language model into multiple smaller agents improves its ability to review financial credit memos. The authors compare one big model checking all tasks at once to a team of smaller agents each doing part of the task. They find the team catches slightly more defects but also raises more false alarms, and uses more computing power. Showing the smaller agents the tasks they were not assigned reduces false alarms. Trying multiple single-agent runs can outperform the team. Overall, using multiple agents has small trade-offs depending on instructions and model type.
Open → 2610.27488v1

Comparison of metrics for summarizing long financial reports with transformers

A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers

Abstract: There are more than 2,000 listed companies on the UK's London Stock Exchange, divided into 11 sectors who are required to communicate their financial results at least twice in a single financial year. UK annual reports are very lengthy documents with around 80 pages on average. In this study, we aim to benchmark a variety of summarisation methods on a set of different pre-trained transformers with different extraction techniques. In addition, we considered multiple evaluation metrics in order to investigate their differing behaviour and applicability on a dataset from the Financial Narrative Summarisation (FNS 2020) shared task, which is composed of annual reports published by firms listed on the London Stock Exchange and their corresponding summaries. We hypothesise that some evaluation metrics do not reflect true summarisation ability and propose a novel BRUGEscore metric, as the harmonic mean of ROUGE-2 and BERTscore. Finally, we perform a statistical significance test on our results to verify whether they are statistically robust, alongside an adversarial analysis task with three different corruption methods.

Wed 7 OctComputation and Language
The gist
Long financial reports from UK companies are very detailed and hard to read quickly. The authors studied different ways to make short summaries using advanced language AI models called transformers. They also looked at various scoring methods to see which best measure how good the summaries are. They introduced a new combined score, BRUGEscore, that mixes two popular scores to better capture summary quality. Finally, they tested if their results are reliable and checked how the methods perform when the input reports are deliberately changed.
Open → 2610.09529v1

Topic model improves understanding of related sentence meanings in text

vMF Sentence LDA: A Spherical Topic Model over Sentence Embeddings

Abstract: Latent Dirichlet Allocation (LDA) and models derived from it remain widely used topic models. LDA observes each document as a bag-of-words and models each topic by a categorical distribution over the vocabulary, so that it uses neither the internal structure of the document nor the similarity in meaning between words. Earlier work has responded to this limitation in two ways: many models have introduced embeddings, and some have assigned topics to sentences rather than to words. Their combination, a topic model that observes sentence embeddings, remains little explored. We propose vMF Sentence LDA (vSLDA), which keeps the admixture structure of LDA, observes each sentence as its L2-normalized embedding and models each topic by a von Mises-Fisher (vMF) distribution, which matches the cosine geometry of sentence embeddings. Its per-topic parameter count and per-iteration cost are linear in the embedding dimension, versus quadratic for the full-covariance Gaussian distribution in the existing model over sentence embeddings. We evaluate vSLDA where the limitation is expected to matter most, among topics that share much of their vocabulary: the topics that subdivide the one subject of a collection, and the narrow topics that result when a corpus is divided into a large number of topics. On two corpora, vSLDA attains the best mean rank against eight baselines when the fine categories within each coarse category are classified from the document-topic distributions. On the whole corpus, its advantage appears or widens as the number of topics grows. Weighting the word frequencies of each sentence by its topic posterior yields expected topic-word counts of the same form as those of LDA, so that the standard topic coherence and diversity measures apply to models that assign topics to sentences. On their product, topic quality, vSLDA leads in most within-category conditions of both corpora.

Sun 4 OctComputation and LanguageMachine Learning
The gist
Topic models help computers understand large collections of texts by finding groups of related words. Traditional models treat documents as collections of individual words, ignoring sentence meaning and the similarity of words. The authors introduce a new model called vMF Sentence LDA that groups topics using sentence meanings represented as points on a sphere, which better matches how sentence meaning is measured. Their model improves the ability to distinguish fine categories within topics, especially when topics share many words or when many topics are used.
Open → 2610.05095v1

Veritygate checks if llm claims match structured evidence reliably

VERITYGATE: A Four-Gate Schema-Level Faithfulness Framework and Paired Benchmark for Grounded LLM Narrations over Structured Evidence

Abstract: Fluent LLM explanations may not follow the evidence from a structured system. We present VERITYGATE, a four-gate checker for declared evidence IDs, entities, numbers, and claim types. It checks a fixed schema; it does not verify every fact in the prose. At r=0 and r=1, we test 900 instances per setting (450 grounded-ungrounded pairs) with GPT-4o-mini, Llama-3.3-70B, and Claude Sonnet 4.6. Under this schema-level contract and before repair, 80.3% of mini claims and 47.9% of Sonnet claims fail. These are verifier rejection rates, not prose-hallucination rates. One repair pass raises claim survival from 19.7% to 28.0% for mini and from 52.1% to 54.3% for Sonnet. Verified claims per example change by +0.14 for mini, -0.71 for Llama, and -0.47 for Sonnet, so survival and output volume must be reported together. A second Sonnet pass gives no clear gain. At r=1, Gate 4 covers 97.0%, 98.7%, and 100% of failing claims for mini, Llama, and Sonnet. Small human studies support the rules but show gaps between schema checks and correct prose. A domain-specific GPT-4o judge test shows an order effect, so it is only a usefulness check. We release the code and data.

Wed 30 SeptComputation and LanguageMachine Learning
The gist
Large language models (LLMs) can produce fluent explanations that don't always match the data they are supposed to be based on. The authors created VERITYGATE, a tool that checks certain key parts of these explanations—like evidence IDs, numbers, and claim types—against a structured schema instead of every fact in the text. They tested several LLMs and found many claims failed these checks, but some improvements are possible with automated repairs. Small human tests confirmed the rules mostly work but can't catch everything, and further testing showed that judging the quality of explanations is complex.
Open → 2610.00833v1

Source-preserving alignment improves locating evidence text in scientific pdfs

Source-preserving alignment for robust evidence localization in scientific PDFS

Abstract: Scientific information-extraction systems often return a claim with an evidence string, which users must locate in the original PDF. This is challenging because the extracted evidence and PDF text layer are different representations: line wrapping, Unicode variants, superscripts, citation markers, and fragmented items alter text sequences and geometry. We present a source-preserving alignment framework: normalize text for robust matching while preserving provenance for accurate localization. It aligns evidence with normalized page text, maps matches back to source-character spans, and renders only their geometry. When exact alignment fails, line-break-aware token alignment recovers supported spans while excluding unmatched noise. Experiments on 1,020 chemistry papers show that the framework achieves a 92.6\% quote-level automatic localization rate, compared with 43.6\% for text search and 19.1\% for a precomputed bounding-box baseline. Component ablation confirms distinct contributions from normalization and approximate token alignment, while human verification assesses the visual correctness of returned highlights. Overall, these results demonstrate that reliable evidence verification requires robust matching and precise localization within a shared source-preserving alignment representation.

Mon 28 SeptArtificial Intelligence
The gist
Finding exact evidence in scientific papers is hard because the text displayed can differ from how it’s stored, with line breaks and special characters causing confusion. The authors present a new way to match pieces of evidence text back to their original spots in PDFs, keeping track of where the text came from while allowing some flexibility in matching. This method works much better than simple text search or older techniques, especially on chemistry papers. It helps verify scientific claims by showing exactly where in a paper the supporting text is found.
Open → 2609.35588v1

EviStreams lets medical teams control AI data extraction for reviews

EviStreams: Human-in-the-Loop AI Data Extraction for Systematic Reviews in Medicine

Abstract: Systematic reviews underpin clinical guidelines, yet their data-extraction step is a major expert-labor bottleneck bound by a protocolized workflow: two reviewers extract each study independently, an adjudicator resolves disagreements, and the team keeps an auditable record of how every value was produced. Large language models can assist with extraction, but that assistance must fit established review protocols and preserve reproducibility. We present EviStreams, a live, open-source, no-code web platform that puts review teams in control of AI-assisted extraction at three key stages: program design (a structured decomposition approved before any code runs), field specification (typed field definitions calibrated from a pilot), and extracted predictions (reviewer-blinded dual review with adjudication). Working through a form builder, a domain expert defines typed fields rather than prompts, runs extraction over uploaded PDFs, inspects every value alongside the supporting passage it came from, and resolves a reviewer-blinded dual review into an auditable consensus export. An evaluation across four clinical corpora and three frontier model families, released with the system, shows that extraction quality is shaped far more by the field specification than by the choice of model. EviStreams is live at https://evistreams.com/demo and released under Apache-2.0.

Wed 23 SeptComputation and Language
The gist
Systematic reviews in medicine require experts to carefully pull data from many studies, which is time-consuming and must follow strict rules. The authors present EviStreams, a web platform that helps review teams guide AI in extracting data while keeping human oversight and follow protocols to ensure accuracy. Experts use a form builder to define data fields, review AI-suggested values alongside evidence, and resolve any disagreements anonymously for a reliable final dataset. They found that how fields are defined impacts extraction quality more than which AI model is used.
Open → 2609.27418v1

Clinical experts struggle to guide large language models in data extraction

"I Know Where to Look," But Does the LLM? Charting the Gaps Between Clinical Expert Needs and Unstructured Data Abstraction Tools

Abstract: Clinical data abstraction, the process of distilling structured information from patient records, plays a key role in advancing knowledge about diseases such as cancer. Information extraction (IE) with large language models (LLMs) could accelerate this process, but it is unclear whether current frameworks effectively support clinical researchers without AI expertise. To address this, we co-designed an interactive LLM-based abstraction system called Libretto with seven cancer research teams, then evaluated the system's ability to help them answer real-world research questions. We found that while clinicians knew where and how to annotate complex concepts in patient notes, in twelve of fourteen tasks they faced barriers to replicating those intuitions with LLMs. Contextual note reliability judgments, difficulties in steering vibe-coded prompts, and inflexible evaluation strategies necessitated fundamental changes to the IE workflow. Our results highlight open problems for HCI research to bridge the gaps between AI data work tools and clinical users' needs.

Wed 16 SeptHuman-Computer Interaction
The gist
Clinical researchers need to pull important facts from complex patient records to study diseases like cancer. The authors created a tool using large language models to help with this, but found that doctors often have tough times teaching the AI what details matter and how to extract them accurately. Challenges included judging which notes are reliable and tailoring the AI prompts. The study shows that current AI tools don’t fully match what clinical experts need and points out areas that need improvement.
Open → 2609.19318v1

Refverifier aids peer reviewers in checking scientific citations accurately

RefVerifier: Semi-Automated Reference Claim Verification for Scientific Manuscripts

Abstract: As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing.

Mon 7 SeptSoftware Engineering
The gist
Scientific papers often include claims supported by citations, but verifying these citations takes a lot of time, which reviewers may not have. The authors created RefVerifier, a tool that helps reviewers check if a paper’s claims match the cited sources by automatically finding and summarizing the relevant parts of cited papers. RefVerifier can find citation information, retrieve related open-access papers, and highlight evidence passages, helping reviewers make faster and more accurate judgments on citation reliability. This approach does not replace reviewers but assists them by automating some tedious steps.
Open → 2609.07652v1