LLM simulated jurors show bias based on defendant background and ideology
When Does Defendant Statement Matter? A Study of Bias and Persuasion in LLM-Simulated Jurors
Computation and LanguageComputers and Society
Summary
The paper investigates how large language models (LLMs) act like jurors in criminal cases and how defendant statements influence their decisions. The authors created a benchmark called JuryBench with cases where defendants give different statements and come from various backgrounds. They found that emotional statements can sometimes backfire, and jurors tend to be harsher to defendants from different backgrounds while being more lenient to those who share their background. Juror political ideology also strongly affects verdict severity. These insights help understand both the potentials and limitations of using AI to simulate jury decisions.
What this means in practice
- •For legal technology developers: Develop AI tools that simulate juror decisions to help analyze courtroom strategies and potential biases in verdicts on criminal cases.
- •For policy analysts: Use AI-simulated juror behavior models to study and improve jury instructions and decision-making fairness considering juror biases.
Authors
Cho-Ying Wu
Abstract
LLMs have been used to simulate human decision-making in professional settings, yet their behaviors in common-law jury trials remain unexplored. We study when and how a defendant's courtroom statement affects LLM-simulated jurors, focusing on persuasion, ideological bias, and background-based affinity. To support the analysis, we introduce JuryBench, a benchmark containing controversial criminal cases in U.S. criminal law. In each case, a defendant can claim various plausible justifications to support acquittal or reduced liability. We fix the base case and design defendants of different backgrounds, who give courtroom statements with varying emotional appeal or rebuttal. Jurors with diverse ideological profiles across the spectrum are simulated. We examine 20 frontier LLMs, resulting in a total of 432K decisions and rationales, and quantify changes in verdict severity. Our findings show that LLM-jury simulation echoes many human-jury findings. First, emotional persuasion can be detrimental, since jurors may perceive it as evidence of guilt or inconsistency. Next, we show that background fit between jurors and defendants is a stronger and significant factor than other isolated factors, and that jurors are in general harsher toward opposite-background defendants and lenient toward same-background ones. Finally, we find that juror ideology also strongly shapes severity judgments. These findings highlight both the promise and risks of using LLMs to model jury reasoning and call for careful evaluation. The data and code are available at https://github.com/choyingw/JuryBench