Thai legal question answering models struggle with law changes over time
NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains
Computation and Language
Summary
Legal rules can change a lot over time, but many legal AI systems only check answers against current laws, not the laws in force when a court decision was made. The authors studied Thai court decisions and found that many refer to older versions of law that have since changed a lot. They created NitiBench+, a tool that lets AI answer legal questions while considering how the law changed over time. Their tests show that AI models perform worse if they use only current law texts instead of the law at the decision time, highlighting the need for legal systems to track law dates carefully.
What this means in practice
- •For legal tech developers: Build AI tools that answer Thai legal questions accurately by accounting for law versions valid at decision dates.
- •For regulatory compliance teams: Ensure automated legal compliance checks consider historical law changes by integrating temporal legal benchmarks like NitiBench+.
Authors
Pawitsapak Akarajaradwong, Sarana Nutanong, Chompakorn Chaksangchaichot
Abstract
Law changes over time. A court decides a case under the law in force on that day, and legal QA benchmarks grade answers against today's law. We checked 156,650 decisions of four Thai courts. Up to 67% of them cite law that has since been amended or repealed. We introduce NitiBench+, the sequel of our NitiBench, a Thai legal QA benchmark with 1,670 items across 17 domains, built from decisions of three courts. For each cited section it holds both the text in force on the decision date and the current text, and its retrieval corpus holds every dated version of every Thai statute section. Answers are graded on the Thai Bar Council rubric, the Bar-standard score, and on the overlap between the cited sections and the court's. The report has two parts. Part I measures the cost of the wrong text on 13 open-weight models. Reading the current text of amended law drops the Bar-standard score by 0.25 and missing repealed law drops it by 0.79. The penalty comes from rule changes, 0.45 on the 428 rule-changed items and none on the 331 wording-only items, and Citation F1 does not see it. The text in force is the fair setting. Part II benchmarks today's models on that setting. A public leaderboard ranks 26 models on a held-out test split of 500 items stratified by domain, with a validation split of the same shape released for public use, from memory alone, with the court's sections provided, and as a retrieval agent over the law in force, with the cost per item and the hallucinated citations. The best model scores 7.78 out of 10 and the best open-weight model 6.39. Without a search tool only five open-weight models recall any Thai law, and with it twenty of the 26 models gain and six lose. Leaderboard will take new models as they are released. Legal benchmarks must track dates, and on the right text today's models still leave a wide gap on Thai law.