Calibrating large language model judges for human and ai conversations
Calibrating LLM Judges for Human and AI Conversations
Human-Computer Interaction
Summary
Judging how well conversations go is hard, even for people. The authors studied how well advanced language models (LLMs) judge conversations between humans and AI. They found that scoring each conversation alone works okay, but comparing two conversations directly has problems with long chats and biases. To fix this, they made a simple way to adjust the LLM scores so they can be compared fairly. They also created a new set of conversations to test these judges and showed their method helps even when judging different kinds of talks.
What this means in practice
- •For ai chatbot developers: Improve evaluation of chatbot conversations by calibrating diverse judge models onto a common scale for better comparison.
- •For customer service teams: Use calibrated judgment models to assess and enhance conversations with AI agents across various interaction scenarios.
Authors
Maike Züfle, Patrícia Schmidtová, Vilém Zouhar, Shree Harsha Bokkahalli Satish, Erica Cooper, Shobhit Banga, Vaibhav Nalawade, Manmeet Kaur, Jan Niehues, Markus Müller, Ondřej Klejch
Abstract
Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias. Since this leaves judge scores incomparable across models, we propose a small anchor set and a calibration function that calibrates any judge onto a shared, interpretable scale. We further release the Voice Arena Goal Dataset (VA), 200 task-oriented human-AI and human-agent conversations with pairwise annotations, revealing a substantial gap between current judges and human-level discrimination. Using VA, we test whether CANDOR-fitted calibration transfers to human-AI conversations, finding it brings judges onto a shared scale despite never observing VA during fitting.