Automatic method maps how scientific claims vary in generality
How broad is that claim? Mapping Generalisation in NLP Research
Computation and LanguageArtificial Intelligence
Summary
Scientific papers often make claims that range from very specific to broadly general, which can sometimes be misleading. The authors developed a detailed way to label these claims by how general they are, helping to spot when researchers might overstate their findings. They created a computer system that uses large language models to automatically sort scientific sentences into categories based on generality. They checked this system against human judgments and used it to label many natural language processing research papers to understand how generalizations are used and linked to things like citations and vague wording.
What this means in practice
- •For research paper editors: Identify and highlight overly broad claims in submitted NLP manuscripts to improve scientific accuracy and clarity.
- •For academic publishers: Automatically classify and tag the level of generalisation in NLP articles to assist reviewers and readers in assessing claim strength.
Authors
Chenxin Diao, Nataliya Stepanova, Emily Allaway
Abstract
Generalisations are common in scientific communication, even though they are semantically ambiguous. An automated method is needed to identify and categorise claims according to their level of generalisation, in order help detect an over-reliance on generalisations and possible misrepresentations of scientific findings. We introduce a comprehensive taxonomy of generalisations in the scientific domain, NLPGenX, which labels claims according to their level of generality and framing within the text. We operationalise this taxonomy with an LLM-powered framework, NLPGenA, that automatically classifies sentences from scientific articles into 5 different generalisation classes. We validate our framework with human annotators and use the framework to construct a large-scale dataset of NLP papers annotated according to generality, with auxiliary labels for hedging and vague descriptors (NLPGens). We use NLPGens to analyse the use of generalisations in NLP papers across multiple venues and subdomains, and to examine associations with citation counts, hedging, and vague descriptors.