Large language model agents improved with professional fact-checking skills
Standing on the Shoulders of Giants: Enhancing LLM Agents for Automated Fact-Checking with Human Expert Skills
Artificial IntelligenceComputation and Language
Summary
Automated fact-checking tools using large language models still struggle to find and verify evidence well. The authors created a special skill module based on real expert fact-checking guidelines to help these tools work better. By letting the language models access this skill during their checking process, the system, called SkillCheck, improves accuracy while reducing computer costs. This approach helps language models search smarter and examine evidence more carefully without needing extra training or constant human help.
What this means in practice
- •For fact-checking teams: Use SkillCheck to enhance automated verification by integrating expert fact-checking guidance during the evidence search and claim review process.
- •For newsroom automation teams: Deploy SkillCheck to help automate detecting misinformation more accurately and cost-effectively in large-scale media monitoring workflows.
Authors
Xinwen Chen, Haorui He, Yupeng Li
Abstract
Automated fact-checking (AFC) systems powered by large language model (LLM) agents still face challenges in evidence retrieval and claim verification. Existing approaches improve their capabilities through computationally costly post-training or human-in-the-loop feedback that is difficult to scale. To transfer professional fact-checking expertise to LLM agents efficiently, we distill publicly available guidelines published by Agence France-Presse (AFP) Fact Check into a reusable, modular Professional Fact-Checking Skill (PFC-Skill). We then present SkillCheck, the first AFC system that gives LLM agents dynamic access to PFC-Skill to guide evidence retrieval and claim verification at inference time. Extensive experiments demonstrate that SkillCheck improves upon skill-free DEFAME baselines across multiple LLM backbones, with gains of 2.2--4.8 pp. in Accuracy and 1.5--6.3 pp. in Macro-F1 on the AVeriTeC benchmark. Notably, SkillCheck with GPT-4o-Mini exceeds DEFAME with GPT-4o by 3.4 pp. in Accuracy while cutting inference cost by roughly 61%. The performance gains also extend to open-web retrieval. Ablations show that incorporating the skill during evidence retrieval provides most of the gains, while incorporation into both stages yields the best overall performance. Case studies show that PFC-Skill enables more targeted search and closer scrutiny of evidence. To conclude, our work shows that LLM agents can "stand on the shoulders of giants" by drawing on the best practices of professional fact-checkers, substantially enhancing AFC performance without extra training or continuous human supervision.