Papers for

academic publishers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Automatic method maps how scientific claims vary in generality

How broad is that claim? Mapping Generalisation in NLP Research

Abstract: Generalisations are common in scientific communication, even though they are semantically ambiguous. An automated method is needed to identify and categorise claims according to their level of generalisation, in order help detect an over-reliance on generalisations and possible misrepresentations of scientific findings. We introduce a comprehensive taxonomy of generalisations in the scientific domain, NLPGenX, which labels claims according to their level of generality and framing within the text. We operationalise this taxonomy with an LLM-powered framework, NLPGenA, that automatically classifies sentences from scientific articles into 5 different generalisation classes. We validate our framework with human annotators and use the framework to construct a large-scale dataset of NLP papers annotated according to generality, with auxiliary labels for hedging and vague descriptors (NLPGens). We use NLPGens to analyse the use of generalisations in NLP papers across multiple venues and subdomains, and to examine associations with citation counts, hedging, and vague descriptors.

Sun 13 SeptComputation and LanguageArtificial Intelligence
The gist
Scientific papers often make claims that range from very specific to broadly general, which can sometimes be misleading. The authors developed a detailed way to label these claims by how general they are, helping to spot when researchers might overstate their findings. They created a computer system that uses large language models to automatically sort scientific sentences into categories based on generality. They checked this system against human judgments and used it to label many natural language processing research papers to understand how generalizations are used and linked to things like citations and vague wording.
Open → 2609.14770v1

Large language models struggle with detailed novelty checks of AI papers

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

Abstract: Large language models (LLMs) are increasingly used in peer review at major AI conferences, yet novelty remains a persistent weak point. Existing benchmarks assess novelty as a single holistic score, making it difficult to diagnose which dimension a model misjudges or whether its evidence is faithful. We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis. The benchmark contains 619 paper pairs and 50 multi-paper sets, drawn from two expert sources: ICLR reviewer overlap claims and survey co-citations. Instances are independently labeled along three dimensions: task, problem, and method, capturing application goals, technical challenges, and solution approaches. We propose a cascading diagnostic pipeline that verifies per-dimension correctness, evidence grounding, and logical support. Evaluation of 18 LLMs shows hallucination rates ranging from 0% to 39% across dimensions, and among non-hallucinated correct-positive judgments, over 70% cite evidence fails to logically support the stated reason. The best-performing model, GPT-5.5, achieves 43-72% Verified F1 across dimensions, while most models retain less than half of their raw F1 after faithfulness verification. These results suggest that current LLMs remain far from reliable scientific novelty assessment, particularly when correctness is conditioned on faithful evidence grounding.

Thu 10 SeptArtificial Intelligence
The gist
Assessing how new or original a scientific paper is can be tricky for large language models (LLMs) that help with AI peer review. The authors created NovGauge, a detailed test set that breaks down novelty into parts like tasks, problems, and methods, to see exactly where models fail. They found many models either hallucinate or give reasons not supported by evidence. Even the best model only gets partial scores when checked carefully, showing that current AI tools are still unreliable at judging scientific novelty.
Open → 2609.11234v1