LLJ Cards guide best practices for using AI as evaluators
LLJ Cards: Best practices for the Use of LLMs as Judges
Computation and Language
Summary
Using artificial intelligence programs, called large language models (LLMs), to judge or evaluate things is becoming popular because they can work fast and cheaply. But people have noticed that these AI judges don’t always provide reliable or fair evaluations. The authors created a set of practical guidelines called LLJ Cards to help people use AI judges more carefully. These guidelines aim to make sure evaluations are valid, reliable, and easy to check by others.
What this means in practice
- •For ai development teams: Design evaluation processes for AI outputs that follow structured guidelines ensuring consistency and transparency in automated judgments.
- •For software quality assurance teams: Adopt standardized principles from LLJ Cards to improve automated review reliability in software testing environments.
A position paper. It proposes an approach and reports no results.
Authors
Khaoula Chehbouni, Melina Medjdoub, Florian Carichon, Golnoosh Farnadi, Jackie Chi Kit Cheung
Abstract
In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJs), these systems have been widely adopted by researchers and practitioners across a broad range of measurement tasks, driven by their strong performance, scalability, and cost-effectiveness relative to human judgment. However, a growing body of work has shown that the use of LLJs raise concerns about their validity and reliability as evaluators. Existing efforts to address these challenges have largely focused on developing bias-mitigation techniques and refining prompting strategies. While these approaches represent an important step forward, they primarily offer technical fixes and leave a more fundamental challenge unaddressed: the lack of standardized, transparent, and reproducible evaluation practices. In this paper, we introduce LLJ Cards, a framework that synthesizes best practices from measurement theory, natural language generation, and machine learning literature into practical guidelines for LLJ-based evaluations. While LLJs offer a promising path toward scalable evaluation, their effective use requires grounding in rigorous evaluation principles to ensure validity, reliability, and reproducibility. LLJ Cards addresses this need by providing a structured framework for applying these principles in the design and reporting of automated evaluations.