Agora method improves aggregating opinions of large language model judges
AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory
Artificial IntelligenceComputation and Language
Summary
When many AI models called large language models (LLMs) act as judges for tasks usually done by humans, their opinions can vary in accuracy. The authors propose a way to weight each AI judge’s opinion based on reliability, learned without needing examples judged by humans. Their method, AGORA, puts every task on a quality scale and figures out how sensitive and trustworthy each judge is by looking at all tasks together. Tests show AGORA works better especially when the number of tasks is small or judges differ a lot in skill.
What this means in practice
- •For machine learning evaluation teams: Improve the accuracy of AI-generated task quality assessments when human labels are limited and judge reliability varies widely.
- •For automated content moderation teams: Enhance trustworthiness of multi-AI moderation decisions by weighting AI judges’ opinions based on learned reliability without relying on large human-labeled datasets.
Authors
Junhyung Ahn
Abstract
LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. Judge pools mix judges of very different reliability, while the human-labeled sets that validate them hold a few hundred items or fewer. In this regime the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items. We propose AGORA (Aggregation Of Reliability-weighted Assessments), an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned from all items jointly and without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items. It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps, and its learned trust weights track per-judge human validity.