Automated systems learn expert judgment from reasoning to judge AI news quality
Distilling Expert Judgment at Scale
Artificial Intelligence
Summary
Judging high-stakes topics like healthcare or politics often needs experts’ careful thinking, which is hard to do automatically. The authors developed a method that collects how experts think and use those rules to create automated judges that evaluate information continuously. They tested this idea on AI news articles and found their automated system closely matched expert opinions, doing much better than simply prompting AI models without expert input. This approach could help build reliable evaluation tools where expert knowledge is rare but important.
What this means in practice
- •For media quality teams: Automatically assess news source quality and neutrality by applying expert reasoning in editorial standards for continuous monitoring.
- •For financial compliance officers: Use expert-calibrated automated evaluation to monitor and judge complex financial reports accurately and at scale where expert review is limited.
Tested on one dataset.
Authors
Robbie Goldfarb, Andrew B. Hall, Jillian Fisher, Kathryn Salam, Matt Wilde
Abstract
Frontier AI models are increasingly deployed on topics where the stakes are high and expert judgment is required, including healthcare, defense, finance, governance, and politics. We present a general methodology for distilling expert human judgment into automated evaluation systems that operate at scale. Our approach assembles networks of domain experts and develops techniques for extracting not just their conclusions but their reasoning processes, encoding those processes into structured editorial standards that automated judges can apply continuously. We ground our approach in a long tradition of research on the nature of expertise and the conditions under which expert judgment is reliable and more effective than other approaches, and argue that independent, expert-calibrated evaluation is necessary not only on epistemic grounds but as a matter of accountability and public trust. We validate the methodology on AI news reporting as a first application domain. On source quality, two independent domain experts achieve Krippendorff's $α= 0.83/0.69$ on a held-out test set when applying the editorial standards, and our automated judge reaches $F1 = 0.81/0.97$ against expert consensus. On neutrality, experts achieve $α= .72$ on the test set, and the automated judge reaches $F1 = 0.86$ against expert consensus. Additionally, a controlled comparison shows that expert-calibrated judges substantially outperform uncalibrated frontier models on every source-quality metric, demonstrating that encoding expert reasoning is the critical step that prompt engineering alone cannot replace. We present this benchmark as a proof of concept for a broader evaluation infrastructure applicable to any domain where judgment matters and expertise is scarce.