Veritygate checks if llm claims match structured evidence reliably
VERITYGATE: A Four-Gate Schema-Level Faithfulness Framework and Paired Benchmark for Grounded LLM Narrations over Structured Evidence
Computation and LanguageMachine Learning
Summary
Large language models (LLMs) can produce fluent explanations that don't always match the data they are supposed to be based on. The authors created VERITYGATE, a tool that checks certain key parts of these explanations—like evidence IDs, numbers, and claim types—against a structured schema instead of every fact in the text. They tested several LLMs and found many claims failed these checks, but some improvements are possible with automated repairs. Small human tests confirmed the rules mostly work but can't catch everything, and further testing showed that judging the quality of explanations is complex.
What this means in practice
- •For ai system developers: Automatically check that AI-generated claims correctly reference structured evidence before releasing outputs.
- •For legal document reviewers: Support verification that AI-generated legal narratives align with referenced case facts and documents using schema checks.
Authors
Sachin Gupta
Abstract
Fluent LLM explanations may not follow the evidence from a structured system. We present VERITYGATE, a four-gate checker for declared evidence IDs, entities, numbers, and claim types. It checks a fixed schema; it does not verify every fact in the prose. At r=0 and r=1, we test 900 instances per setting (450 grounded-ungrounded pairs) with GPT-4o-mini, Llama-3.3-70B, and Claude Sonnet 4.6. Under this schema-level contract and before repair, 80.3% of mini claims and 47.9% of Sonnet claims fail. These are verifier rejection rates, not prose-hallucination rates. One repair pass raises claim survival from 19.7% to 28.0% for mini and from 52.1% to 54.3% for Sonnet. Verified claims per example change by +0.14 for mini, -0.71 for Llama, and -0.47 for Sonnet, so survival and output volume must be reported together. A second Sonnet pass gives no clear gain. At r=1, Gate 4 covers 97.0%, 98.7%, and 100% of failing claims for mini, Llama, and Sonnet. Small human studies support the rules but show gaps between schema checks and correct prose. A domain-specific GPT-4o judge test shows an order effect, so it is only a usefulness check. We release the code and data.