Legal ai system verifies every claim against cited sources
GANDR: Claim Auditing for Verifiable Legal Answer Generation
Computation and LanguageArtificial IntelligenceInformation Retrieval
Summary
Legal answers from AI can be hard to trust if the system’s citations don't actually back up its claims. The authors created GANDR, a system with two parts: one writes legal answers carefully, and another checks every claim to make sure it matches its source exactly. This detailed checking improves accuracy and helps flag when claims are not properly supported. Their system outperforms existing methods on a legal benchmark.
What this means in practice
- •For legal tech developers: Build AI tools that deliver verifiable legal answers by auditing each claim against trusted citations in legal texts.
- •For compliance monitoring teams: Use claim auditing to detect unsupported or fabricated information in generated legal documents or summaries.
Authors
Chen Qian, Yimeng Wang, Yu Chen, Lingfei Wu, Andreas Stathopoulos
Abstract
In high-stakes domains such as legal practice, a language-model answer is only useful to the extent that a reader can verify each claim against the source the system cites. Current grounded-generation pipelines score the answer as a whole, so a correct conclusion can rest on fabricated or loosely matched citations and still score well. Closing this gap requires both a system built for per-claim verification and an evaluation that measures it. We introduce GANDR (Grounded ANswer DRafter), a two-agent system in which a Drafter writes an answer in a structured legal-reasoning format and a separate Critic, with the same view as a human verifier, audits each claim against its cited source and emits a per-claim audit trace on every round. We pair it with a strict correctness criterion requiring every citation to resolve to a passage the retriever returned. On a 185-item legal benchmark where all six systems share one backbone, one retrieval surface, and one citation instruction, GANDR ranks first on every primary metric, reaching 70.8% strict accuracy and leading the strongest baseline by 11.3 points (p<0.01). Reverting the protocol-anchored commit rule lowers strict accuracy by 22.7 points, and the strict lead stays positive on three further backbones, at +3.2 to +6.5 points. This lead traces to the Drafter configuration and the protocol-anchored commit, not to rewriting. Against two law-trained annotators the audit flags under-supported claims at F1 0.84 as a binary detector, while its four-way verdict labels agree only weakly and are advisory. Code is available upon request.