Papers for

fact-checking tool builders

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Factuality metrics face trust issues in judging language models

Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation

Abstract: Evaluating the factual correctness of large language models (LLMs) is vital for many applications. But are our evaluation tools themselves trustworthy? Despite the rise of factuality-based metrics, their sensitivity and reliability remain underexplored. This paper introduces a meta-evaluation framework that systematically tests these metrics using controlled corruptions of gold standard answers. Our method generates ranked outputs with known degrees of degradation to probe how metrics capture nuanced changes in truthfulness. Our experiments reveal that pipeline-based methods, such as the RAGAS's factual correctness metric, better track degradation than LLM-as-judge approaches. We also propose a new variant of the factual correctness metric that provides a competitive and cost-efficient.

Mon 14 SeptComputation and Language

The gist

Checking if language model answers are true is important but tricky. The paper finds that popular methods to judge truthfulness sometimes do not accurately notice small changes in answer correctness. The authors test different judging methods by deliberately corrupting correct answers to see how well the judges detect errors. They found that some step-by-step methods work better than those relying solely on other language models as judges. They also suggest a new, cheaper method that performs competitively.

Open → 2609.15561v1