Large language models struggle with criminal profiling before arrest
Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence
Computation and LanguageComputers and Society
Summary
Figuring out who committed a crime before the suspect is caught is really hard, especially since the evidence is often incomplete. The researchers created a new dataset of real homicide cases from multiple countries to test how well large language models (LLMs) can guess details about suspects from limited evidence. They found that while LLMs can extract clear facts, they struggle with deeper reasoning tasks, like understanding motives or relationships between victims and offenders. The study also showed that these AI models still have big gaps compared to human experts and sometimes show biased judgments.
What this means in practice
- •For law enforcement analysts: Assess AI tools' limits when using language models to help infer suspect profiles from partial crime scene evidence during investigations.
- •For forensic technology developers: Design systems that combine human expertise with AI for improving suspect attribute prediction before arrests, focusing on filling reasoning gaps identified.
Authors
Yutong Yao, Yanjie Cao, Guanhua Chen, Xu Yang, Junchao Wu, Zeyu Wu, Lidia S. Chao, Derek F. Wong
Abstract
Large Language Models (LLMs) are increasingly applied to legal and criminal justice tasks, yet existing work focuses almost exclusively on post-arrest scenarios where the suspect's identity is already known, leaving the critical pre-arrest challenge of inferring suspect characteristics from incomplete evidence largely unexplored. To fill this gap, we introduce the Profiling, Investigation, and Judgment (PIJ), comprising 2,500 real homicide cases from five countries. PIJ evaluates LLMs across three tasks that span the entire criminal investigation pipeline: criminal profiling, which requires abductive reasoning to infer suspect attributes from fragmentary scene evidence, crime process reconstruction, which tests structured information extraction, and sentence prediction, which demands legal deductive reasoning. We evaluate 9 powerful LLMs and find that performance degrades systematically as tasks shift from explicit fact extraction to implicit reasoning over unknown suspect profiles. Categories requiring inferential reasoning, such as motivation and victim-offender relationships, remain the primary bottlenecks. Further analysis reveals substantial gaps between LLMs and human experts, along with pervasive biases in gender, age, and motive attribution. Our findings indicate that pre-arrest inference from incomplete evidence remains an open challenge.