Fact verification improves mainly from better evidence not just answers
What Changes When Fact-Verification Scores Improve? Evidence and Answer Accounting Across Trained Verifiers and LLMs
Computation and Language
Summary
Fact-checking systems give scores based on both the answers they provide and the evidence they use. The authors studied how much better scores come from improving answers versus providing better evidence. They found that improving the evidence has a much bigger effect on the overall score than just improving the answers. Tests with large language models and different datasets showed that giving the models more context also helps the evidence quality more than the answers themselves. This means focusing on better evidence is key to improving fact verification.
What this means in practice
- •For fact-checking system developers: Design fact-verification tools that prioritize evidence quality improvements to achieve higher verification scores.
- •For large language model integrators: Adjust input context size to improve evidence generation quality in automated fact-checking applications using LLMs.
Authors
Han Chen, Yingrui Li
Abstract
A joint fact-verification score assesses answers and submitted evidence together. When the score improves, how much of the gain remains if the answers are held fixed? On FEVEROUS, strict score is the percentage of claims with a correct answer and a complete annotated evidence group in the submitted evidence. Across four trained DeBERTa checkpoints and 7,890 claims, replacing DCUF evidence with UnifEE evidence raises strict score by 9.61 percentage points, compared with 1.96 percentage points in answer accuracy. The paired 95% interval for the strict-score gain is [8.77, 10.43], conditional on these checkpoints. Replacing only the evidence passed to the scorer accounts for 7.92 or 9.08 percentage points when we retain the answers generated from DCUF or UnifEE evidence, respectively. To examine how this evidence gain depends on evaluation choices, we generate 470,400 responses from two 8B LLMs on FEVER, FEVEROUS, and SciFact under two answer formats and two context budgets. Increasing context from 256 to 2,048 tokens raises the fixed-answer evidence gain on FEVEROUS by 3.84 and 3.10 percentage points for Qwen and Llama, respectively. The effects fall short of the prespecified cross-dataset criterion, while some intervals extend beyond the two-point small-effect bound. Post-hoc analyses quantify changes in answers and submitted evidence, and show when aggregate accuracy and evidence-coverage rates miss the claim-level pattern. The four answer-evidence score combinations reveal changes that endpoint and aggregate metrics leave unresolved.