Persian legal question answering improved with graph based retrieval

TUSI: A Citation-Grounded Graph-Based Benchmark

Computation and LanguageArtificial Intelligence

Summary

It's hard to check how well AI tools answer legal questions in Persian because existing tests don't connect questions to real legal documents or cover detailed legal thinking. The authors created TUSI-Bench, a new benchmark based on an organized database of Persian laws and court rulings. This benchmark lets AI systems answer open legal questions with direct references to laws and also tackle multiple-choice questions from actual Iranian Bar exams. They also introduced a retrieval method that uses the relationships in the law data, making it easier to find correct information and improving accuracy by up to 8%.

What this means in practice

  • For legal software developers: Build or evaluate AI systems that provide legally grounded answers in Persian using a realistic and well-structured benchmark of laws and court rulings.
  • For natural language processing teams: Improve retrieval methods in legal AI tools by integrating graph-based relations from the legal knowledge graph to enhance recall and citation accuracy.

Authors

Marzieh Karimi, Mehran Safayani, Alireza Basiri

Abstract

Evaluating retrieval-augmented generation (RAG) systems in Persian law remains challenging because benchmarks lack realism, grounding, and coverage of legal reasoning. Existing resources are either synthetically generated from statutory texts, lacking natural information needs, or derived from informal legal question--answer sources without explicit linkage to authoritative legal documents, making them unsuitable for rigorous evaluation of grounded legal reasoning. To address this gap, we introduce TUSI-Bench, a structured Persian legal benchmark designed for the evaluation of RAG-based systems. TUSI-Bench is built upon a multi-layered legal knowledge framework containing 7,261 legal articles and integrating codified statutes, legal commentaries, doctrinal writings, and Supreme Court unification rulings. We also construct a legal knowledge graph linking articles and concepts for structure-aware retrieval. Based on this foundation, TUSI-Bench provides two complementary evaluation tasks: (i) TUSI-QA, an open-ended question answering benchmark containing 1,999 questions that requires grounded responses with explicit legal citations, and (ii) TUSI-Bar, a multiple-choice benchmark containing approximately 500 four-option questions derived from Iranian Bar examination questions. Furthermore, we propose a graph-based retrieval baseline that leverages relational structure within the legal knowledge graph. Experimental results demonstrate that incorporating graph-based relations improves retrieval recall by up to 8 percentage points compared to standard baselines, highlighting the importance of structured legal representations for RAG systems. The proposed benchmark provides a standardized resource for evaluating grounded legal reasoning in Persian and enables future research on structured legal intelligence.