LLM judges miss task success in simulated user evaluations
GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
Computation and LanguageMachine Learning
Summary
It can be hard to tell which computer assistants do the best job. The authors found that when AI judges rate conversations they have with simulated users, their ratings often don’t match whether the assistant actually completed the task. Even conversations that looked satisfying to human reviewers often failed the tasks. The paper shows this mismatch and suggests a way to check for errors before fully trusting the AI judge’s scores.
What this means in practice
- •For ai product teams: Identify when AI judges’ ratings do not reflect real task success to avoid promoting weaker conversational agents.
- •For chatbot developers: Use a calibrate-then-trust method to catch performance drops through simple checks before relying on automated conversation scoring.
Authors
Umesh Bodhwani, Thanh Tran, Kai Wei
Abstract
Comparing and selecting task-oriented LLM agents increasingly relies on a low-cost offline evaluation gate: persona-driven LLM user-simulators converse with each candidate, an LLM-as-a-judge scores the transcripts, and the higher-scoring agent is promoted. We introduce GAUGE, a reusable offline protocol that measures whether this gate's ranking matches a grounded verifiable reward across 25 agents from six providers on the $τ^2$-bench and SimulatorArena benchmarks, separating two kinds of evaluation validity that release practices conflate: ranking validity and construct validity. First, a satisfaction-success gap: satisfaction carries essentially no information about task success, as conversations rated satisfied by our blind panel are decorrelated from actual success, with 57.5% of them failing the customer's task, a pattern consistent across five rater populations, both benchmarks, and every subjective dimension we rated. Second, while the gate's ranking is robust across the broad capability span, it loses resolution among the near-equal strong agents: this decision-disagreement rate jumps from $<$1% on wide-reward pairs to 31% on close pairs. The gate is thus human-validated yet mis-anchored. As a remedy, we propose a calibrate-then-trust cadence in which a judge-free completion bit is a zero-cost tripwire for truncation regressions.