Large language models outperform traditional methods in scientific claim verification
An Experimental Study of Hybrid Retrieval, NLI and Open-source LLMs for Scientific Claim Verification
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Checking if a scientific claim is true, false, or needs more proof can be hard, especially when the evidence is spread across several sentences. The authors compared two approaches: one that looks at individual sentences (NLI models) and large language models (LLMs) that can consider multiple sentences together. They found that big language models gave much better results than traditional models at deciding if claims are supported or contradicted by scientific texts. However, trying to combine both methods didn’t improve accuracy and sometimes made it worse.
What this means in practice
- •For medical data teams: Use large language models to check if clinical research claims are supported by evidence in medical texts, improving fact verification in healthcare.
- •For scientific information services: Develop tools that retrieve and verify scientific claims more accurately by leveraging combined semantic and lexical search with large language models.$Commercial implications: Enables commercial fact-checking and information validation products for scientific publishers and databases.
Authors
Lakshmi Narayanan P, Narenkarthik K, Naren Kumar C
Abstract
Claim verification is the task of deciding whether a source text supports a claim, contradicts it, or lacks sufficient evidence. Natural Language Inference (NLI) models can judge whether a single sentence supports or contradicts a claim, but they struggle when the evidence is spread across several sentences. Conversely, Large Language Models (LLMs) can reason across sentences, but may draw conclusions the text does not support. In this experimental study, we test whether a conservative fusion rule, labeling a claim Support or Contradict only when both models agree and defaulting to Not Enough Evidence otherwise, combines their strengths. We construct a benchmark of 2,035 claims from SciFact and CliniFact (858 Support, 152 Contradict, 1,025 Not Enough Evidence). We retrieve the top five most relevant sentences per claim using a hybrid of semantic and lexical (BM25) retrieval, which slightly outperforms either method alone. These retrieved sentences are then provided to every verification model. We then compare a DeBERTa-v3-base-mnli-fever-anli NLI baseline against Llama 3.1 8B, Qwen3 8B, and Mistral 7B. Qwen3 achieves the highest Macro-F1 (0.7937), followed by Llama 3.1 (0.7204) and Mistral (0.6388), while the NLI baseline reaches only 0.3157. Contrary to our hypothesis, fusion degrades performance, dropping Macro-F1 to 0.7392, 0.6730, and 0.5991, respectively. To understand this decline, we analyze agreements, disagreements, and common error patterns between the NLI and LLM verifiers. Our error analysis shows that NLI and LLM disagreements often result in correct LLM predictions being overturned.