Legal ai models name laws without truly following them
Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness
Artificial IntelligenceComputation and LanguageComputers and Society
Summary
Large language models explain legal decisions by citing specific laws or cases as if those authorities led to their conclusions. The authors tested whether changing the cited law actually changes the model’s verdict. They found that while models often correctly name the legal authority they’re supposed to rely on, their decisions don’t consistently depend on the cited laws. This means the name of a law is often just a label and not a true explanation, raising concerns about trusting these model justifications in legal contexts.
What this means in practice
- •For legal tech teams: Improve auditing tools to detect when AI legal justifications are superficial and do not truly rely on cited laws.
- •For contract review teams: Enhance contract analysis systems by verifying that AI explanations correspond to actual reasoning rather than placeholder citations.
Authors
Saisab Sadhu, Shreeyans Arora, Pratinav Seth
Abstract
Large language models increasingly justify legal decisions by naming the statute or precedent behind a verdict, treated as evidence that the decision follows from it. We test this directly: holding case facts fixed, we substitute the named legal authority for an unrelated one and decode a model's evolving verdict from its hidden states. Across seven open-weight models (8B-70B) and four benchmarks spanning judicial and contractual reasoning, when explicitly required to justify a verdict by naming the governing authority, models name the correct one in 66.7%-100% of generations, while the verdict changing when the authority changes is far less consistent: 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. Neither scale nor a purpose-built legal-reasoning model (a best-effort LoRA reproduction; Section 6) closes this gap. A red-teaming evaluation on five core models finds compliance with an adversarial instruction hidden in the case facts (73.3%-96.4%) exceeds verdict-swap sensitivity by a wide margin, holding without exception across model rankings. Naming a legal authority is thus a poor proxy for a verdict's dependence on it, while the same verdict remains separately vulnerable to adversarial manipulation. Both findings replicate across checks ruling out prompt-wording noise and confounded sampling, and bear directly on the use of generated legal explanations as compliance or audit artefacts.