Criminal court decisions transformed into structured data for legal research
Unlocking Criminal Court Decisions: Flexible Information Extraction for Empirical Legal Research
Computation and LanguageArtificial Intelligence
Summary
Legal decisions in criminal cases often describe what happened, but turning these descriptions into organized facts is hard because the important details aren't always clear ahead of time. The authors created a method using large language models that helps experts explore what information to extract, find the right text pieces, and standardize the results. They tested this on cases about drunk driving and driving under a ban, improving accuracy by refining the instructions given to the system. Their approach also showed that the system's consistent answers don't always mean they're correct, highlighting the importance of clear guidelines.
What this means in practice
- •For legal data engineers: Create structured databases from unstructured criminal court verdicts by iteratively refining extraction rules with expert input.
- •For legal technology developers: Build improved tools for automated extraction of details in specific criminal offenses using adaptable large language model pipelines.
Authors
Tomáš Knap, Ivana Kvapilíková, Jan Černý, Klára Bendová, Jaromír Šavelka, Vojtěch Pour, Jakub Drápal
Abstract
Criminal verdicts contain descriptions of criminal behavior that can support quantitative legal research, but structuring them requires attributes and extraction rules that cannot always be fully specified in advance. We present an LLM-based pipeline that builds a structured dataset through attribute exploration, text span extraction, and value normalization, guided by annotation instructions that legal experts can refine. We evaluate it on two traffic offenses: driving under the influence (DUI) and driving despite a ban (DDB). For DUI, revising the instructions was associated with higher extraction accuracy (from 52% to 70% of correctly extracted values) and higher agreement between human annotators. Repeated runs of the same model agreed with each other more than with the gold annotations, showing that consistent output is not necessarily correct, while disagreement between annotators and between models helped reveal unclear instructions. For DDB, where the instructions defined fixed categories and output formats, agreement was high for most attributes.