Papers for

criminal justice technology teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Framework guides safe and tracked use of AI in critical workflows

MERCI Cards: An LLM Evaluation and Deployment Framework for High-Stakes Domains

Abstract: As LLMs are increasingly deployed in high-stakes professional workflows, engineers and researchers require principled protocols to systematically track, monitor, and improve model performance across deployment cycles. We present a mathematical framework for iterative LLM evaluation and deployment, and demonstrate its application to AI systems used in criminal justice. Our framework formalizes LLM integration in high-stakes, high-risk, and resource-constrained domains across model selection, rubric design, evaluations and deployment via a weighted multi-objective optimization. We demonstrate that MERCI Cards can guide improvements of the system across deployment iterations, direct developer attention toward under-performing areas, and focus user attention on validation and error-correction in the LLM's outputs.

Sat 3 OctArtificial IntelligenceHuman-Computer Interaction
The gist
Using large language models (LLMs) in important areas like criminal justice is risky and complicated. The paper presents a framework that helps people evaluate, monitor, and improve these AI systems carefully over time. This method uses math to balance different goals like accuracy and resource limits, so developers know where to fix problems and users know when to check results. The framework is designed to keep AI use safer and more reliable in sensitive areas.
Open → 2610.04430v1