Large language model decision systems reduce wrong legal citations cost effectively
Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority
Artificial IntelligenceComputation and LanguageComputers and Society
Summary
Checking legal claims automatically needs tools that can decide when they do or don’t find the right authority. The authors test models that don’t just guess answers but give probability scores, which helps them choose to abstain if no correct answer exists. They find decision models are more cautious and cheaper than typical language models, cutting wrong legal references in half when combined. This approach could make legal AI more reliable by focusing on reducing incorrect citations, not just overall accuracy.
What this means in practice
- •For legal technology developers: Create legal AI tools that reduce incorrect citations by using decision models with abstention capabilities.$Commercial implications: This enables development of commercial legal AI software that improves reliability in legal research and citation checking.
- •For risk management teams: Use calibrated decision models to verify legal claims more cautiously, lowering risks from misapplied legal authorities.
Authors
M. Mikail Demir
Abstract
Given the enormous number of claims to be checked, verifiable legal AI requires scalable, cost-effective models that answer the practical needs of law. System One decision models are built on large language models but, instead of generating text, return a calibrated probability for each option. This suits the verification of legal authority, where the question is closed: a given holding, or none of them. We test whether such models abstain when the correct authority is missing. We extend CaseHOLD with an abstention condition in which the correct holding is replaced in 1,670 of 5,314 items, and compare four decision models with six generative LLMs. When no option is correct, generative LLMs still pick one in 38-86% of items; the most cautious decision model does so in 30%. On standard CaseHOLD, a decision model matches the best generative LLM (88.1% vs. 88.2%) at 1/55 of the cost. Accepting that LLM's answer only when a decision model agrees halves its wrong citations, and distilling a decision model's probabilities into Qwen3-8B, without labels, yields a well-calibrated local model. Legal AI should be judged by its wrong citations, not by accuracy alone.