Large language models struggle to verify legal claims accuracy
When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection
Computation and LanguageArtificial Intelligence
Summary
Legal documents often use big language models to help with research and drafting, but these models can sometimes say something that sounds true but isn't backed up by the actual legal source. The authors created a test called PARCEL that checks if a legal claim matches what a court decision really says. They found that even the best models make mistakes by wrongly saying false claims are supported. This test helps spot when language models make these kinds of legal errors.
What this means in practice
- •For legal technology developers: Evaluate the accuracy of legal claims generated by AI systems when citing court opinions to prevent misleading outputs.
- •For compliance teams: Improve verification processes for legal documents by detecting claims unsupported by their cited sources, reducing risk of compliance errors.
Authors
M. Mikail Demir, M. Abdullah Canbaz
Abstract
Large language models are increasingly used in legal research and drafting, but they can still produce claims that sound convincing without being supported by the cited source. We introduce PARCEL, a benchmark for checking whether a legal claim is supported by the underlying authority. Using recent New York State Court of Appeals decisions, we build a dataset of 3,396 parenthetical-style claims labeled as Supported, Refuted, or Not Found. We cast this task as a three-way natural language inference problem and evaluate several state-of-the-art LLMs in a zero-shot setting. Although the strongest models reach up to 0.97 accuracy, the results also show an important weakness: models still incorrectly mark unsupported claims as supported, even when the full opinion text is provided. Across models, missing support is harder to detect than direct contradiction, and fabricated but plausible citations cause the largest drop in performance. Overall, PARCEL provides a practical benchmark for testing claim-level groundedness in legal RAG systems.