Beyond Verdicts: A Graph-Based Analysis of Human and LLM Reasoning in Scientific Fact-Checking
2026-08-24 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors propose a method to compare how large language models (LLMs) and humans reason when checking scientific claims for errors or misinformation. They create a graph that maps out the reasoning steps linking a false claim to the relevant study and any logical mistakes made. This helps identify whether the LLM and humans reach the same conclusion in similar ways or differently but still validly. Testing three LLMs on false biomedical claims, the authors find variations in how often the models give correct verdicts and how closely their reasoning matches humans. Their work helps understand and improve automated scientific fact-checking tools.
large language modelsscientific fact-checkingmisinformationreasoning graphsfallacious reasoningbiomedical misinformationmodel alignmentclaim verificationgraph-based methodsexplanation generation
Authors
Abdul Ghafoor, Muhammad Arslan Manzoor, Yufang Hou
Abstract
Misinformation that cites legitimate papers can be especially harmful when it distorts what those studies actually report. While existing automatic fact-checking systems based on large language models (LLMs) can assess whether a model assigns an Incorrect verdict and can gen- erate explanations for that decision, they typi- cally do not indicate whether the model follows the same reasoning path as human experts or arrives at the verdict through a different but still valid path. In this work, we introduce a graph- based framework (typed reasoning graph) for comparing human and LLM reasoning paths in scientific fact-checking. Building on prior work on fallacious reasoning in biomedical misinformation, MISSCIPLUS (Glockner et al., 2025), we model each explanation as a rea- soning graph that links the false claim to the relevant study context, study findings, fallacy- supporting premises, and fallacy labels. This representation enables one-to-one alignment of human and LLM reasoning at the level of fallacy-specific sub-graphs. For non-human- aligned LLM paths, we validate grounding in the cited study, relevance to the claim, and suf- ficiency for the verdict. Using 84 false claims from MISSCIPLUS, we evaluate GPT-5, Claude Opus 4.7, and Qwen3-32B across prompt and evidence settings. Results show distinct perfor- mance dimensions: Qwen3-32B has the lowest verdict failure rate, GPT-5 the highest human alignment, and Claude Opus 4.7 weak verdict prediction but often valid reasoning in success- ful cases