GRACE dataset improves Canadian legal reasoning with AI
GRACE: Grounded Adversarial Reasoning over Canadian Law
Computation and LanguageArtificial Intelligence
Summary
Legal AI models often struggle with tasks like arguing a position or using multiple laws together, especially for Canadian law which has fewer resources. The authors created GRACE, a Canadian law dataset with nearly 2,000 examples that require reasoning under incomplete information and combining laws. They fine-tuned a small legal AI model showing it performed better when given the right law texts, demonstrating how GRACE can help build better AI tools for Canadian legal reasoning. When the law texts were missing, the model’s performance dropped, highlighting the value of grounding AI in actual legal documents.
What this means in practice
- •For legal tech developers: Create more accurate AI tools that reason with Canadian federal laws by training on GRACE’s grounded, adversarial legal scenarios.$Commercial implications: Enables development of commercial AI legal assistants specialized in Canadian law with improved reasoning and citation abilities.
- •For government legal teams: Use models tuned on GRACE for informed legal analysis and advocacy support reliant on complex statutory interpretation and uncertainty.
Authors
Jiakang Xu, Wantong Huo, Udom Silparcha, Jonathan H. Chan
Abstract
Large language models have shown strong performance across a range of legal tasks, but existing benchmarks rarely evaluate the ability to take and defend a legal position, reason under incomplete information, or synthesize multiple statutory provisions. This gap is particularly pronounced for Canadian law, which remains underrepresented in legal NLP. We introduce GRACE (Grounded Reasoning Adversarial Canadian LEgal examples), a dataset of 1,915 question-reasoning-answer instances grounded in Canadian federal legislation. GRACE covers three reasoning modes: adversarial advocacy, uncertainty, and applied reasoning. We develop a pipeline that partitions raw statutory text, generates scenario-based questions and reasoning, and filters examples through model-free citation verification and LLM-based quality auditing. As a proof of concept, we fine-tune CLeAR-4B (Canadian Legal Adversarial Reasoning), a lightweight model for grounded legal reasoning, and evaluate it against the unmodified Qwen3-4B base model in open- and closed-book settings. CLeAR-4B substantially improves agreement with teacher outputs and statutory citation behavior when the relevant act text is provided, while its grounding degrades sharply when the statute is withheld. These results suggest that GRACE can support the development of lightweight legal models that reason more effectively from supplied statutory text.