Veritygate checks if llm claims match structured evidence reliably

VERITYGATE: A Four-Gate Schema-Level Faithfulness Framework and Paired Benchmark for Grounded LLM Narrations over Structured Evidence

Computation and LanguageMachine Learning

Summary

Large language models (LLMs) can produce fluent explanations that don't always match the data they are supposed to be based on. The authors created VERITYGATE, a tool that checks certain key parts of these explanations—like evidence IDs, numbers, and claim types—against a structured schema instead of every fact in the text. They tested several LLMs and found many claims failed these checks, but some improvements are possible with automated repairs. Small human tests confirmed the rules mostly work but can't catch everything, and further testing showed that judging the quality of explanations is complex.

What this means in practice

  • For ai system developers: Automatically check that AI-generated claims correctly reference structured evidence before releasing outputs.
  • For legal document reviewers: Support verification that AI-generated legal narratives align with referenced case facts and documents using schema checks.

Authors

Sachin Gupta

Abstract

Fluent LLM explanations may not follow the evidence from a structured system. We present VERITYGATE, a four-gate checker for declared evidence IDs, entities, numbers, and claim types. It checks a fixed schema; it does not verify every fact in the prose. At r=0 and r=1, we test 900 instances per setting (450 grounded-ungrounded pairs) with GPT-4o-mini, Llama-3.3-70B, and Claude Sonnet 4.6. Under this schema-level contract and before repair, 80.3% of mini claims and 47.9% of Sonnet claims fail. These are verifier rejection rates, not prose-hallucination rates. One repair pass raises claim survival from 19.7% to 28.0% for mini and from 52.1% to 54.3% for Sonnet. Verified claims per example change by +0.14 for mini, -0.71 for Llama, and -0.47 for Sonnet, so survival and output volume must be reported together. A second Sonnet pass gives no clear gain. At r=1, Gate 4 covers 97.0%, 98.7%, and 100% of failing claims for mini, Llama, and Sonnet. Small human studies support the rules but show gaps between schema checks and correct prose. A domain-specific GPT-4o judge test shows an order effect, so it is only a usefulness check. We release the code and data.