Large language models assessed for legal philosophy understanding
How Do LLMs Understand Legal Philosophy? Diagnosing Legal-Philosophical Reasoning in LLMs
Computation and LanguageArtificial Intelligence
Summary
Understanding how AI models think about law like real judges is important for legal tasks. This paper presents a new test called LePhilBench that checks if large language models (LLMs) grasp legal philosophical ideas and use them when reasoning about laws. The authors collected detailed expert answers on Supreme Court cases to create this test. This helps see how well LLMs can mimic human-like legal thinking based on different views of law.
What this means in practice
- •For legal ai developers: Evaluate and improve AI models’ ability to reason using different legal philosophies on real Supreme Court cases.
- •For legal technology companies: Build more nuanced AI-based tools that reflect varied legal interpretations informed by philosophy.$Commercial implications: Enables development of advanced legal AI products that better emulate judicial reasoning for clients like law firms.
Tested on one dataset.
Authors
Shuyuan Zheng, Yun-seo Lee, Xing Gao, Jialu Wu, Jiabao Hu, Katrina Qiyao Wang, Xinbo Lin, Qianying Liu, Chuan Xiao, Tao Huang
Abstract
Legal reasoning is a fundamental capability for large language models (LLMs) to complete legal AI tasks such as legal judgment prediction. In legal practice, this reasoning is shaped by legal philosophy: a judge's view of the nature and sources of law guides how legal materials are interpreted and applied, so judges working from the same statutes and precedents can reach different judgments. Existing research has largely overlooked whether LLMs can understand legal philosophy and apply it in legal reasoning, an ability we call \textit{legal-philosophical reasoning}. To fill this gap, we introduce LePhilBench, a benchmark for evaluating the legal-philosophical reasoning of LLMs that comprises two benchmarking tasks and a dataset. The two tasks evaluate how well LLMs understand legal philosophy and how well they apply it in legal reasoning, respectively. The dataset is manually and meticulously constructed. Its test set contains 220 U.S. Supreme Court opinions. A law professor designed fourteen annotation questions on legal philosophy, and three law-trained annotators with formal training in jurisprudence answered all fourteen questions for every opinion, yielding 9,240 legal-philosophical labels. LePhilBench takes the first step toward exploring legal-philosophical reasoning in LLMs.