Juryflow improves ai content judging by focusing on disagreements
JuryFlow: Disagreement-Guided Human-in-the-Loop Multi-Agent Evaluation
Computation and LanguageHuman-Computer Interaction
Summary
Judges often disagree when evaluating AI-generated content, and simply voting on the majority can miss important differences. The authors propose JuryFlow, which breaks down responses into smaller claims and uses a panel to judge each claim, focusing on where judges disagree. A human guides the process by resolving selected disagreements, which helps refine evaluation rules that all judges then follow. JuryFlow was tested automatically and showed better agreement with gold-standard labels than usual methods that rely on single judges or majority votes.
What this means in practice
- •For ai product teams: Improve AI content evaluation accuracy by focusing human review on specific disputed claims, reducing labeling effort and increasing trustworthiness.
- •For content moderation teams: Use disagreement-guided evaluation to handle uncertain cases in automated content judgment, helping prioritize human intervention efficiently.
Authors
Mufeng Yang, Junwei Yu, Yepeng Ding
Abstract
Large language models (LLMs) are increasingly deployed as automated judges for AI-generated content, yet a single judge is unreliable and even a panel of judges leaves a hard residue: when judges disagree, majority voting discards the conflict instead of resolving it. We present JuryFlow, a disagreement-guided, human-in-the-loop multi-agent evaluation framework that treats inter-judge disagreement not as noise to be averaged away, but as a precise, claim-level signal indicating where an evaluation is uncertain. JuryFlow decomposes each candidate response into atomic claims, has a panel of heterogeneous judges assign per-claim verdicts, and builds a disagreement graph whose nodes are scored by verdict entropy and whose edges encode structural similarity between claims. A human acts as a structural guide, selecting which disagreement to resolve through a single, minimal intervention rather than re-labeling the response, after which the focal claim is re-evaluated, the correction propagates along graph edges and to historically similar cases, and is crystallized into reusable rubric entries that all judges inherit, making the evaluator progressively self-refining. To enable large-scale, reproducible benchmarking without human studies, we evaluate JuryFlow in an automatic configuration in which focal selection is made by entropy ranking. On MT-Bench and LLMBar, JuryFlow improves agreement with gold labels over single-judge and majority-vote panel baselines, and ablations isolate the contributions of disagreement-targeted re-evaluation, propagation, and rubric induction. We contribute (1) a human-in-the-loop paradigm that recasts the human from labeler to structural guide, (2) the JuryFlow framework operationalizing it through a disagreement graph, focal re-evaluation, and closed-loop rubric induction, and (3) an evaluation protocol with ablations that isolate where the gains originate.