Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
2026-08-06 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors created a new way to check if tests used to judge task-based chatbots are good or not. Their method uses large language models to act like judges, checking if the tests are consistent, challenging, and cover enough different chatbot behaviors. They showed that their method agrees well with human reviewers and works on both computer-made and human-made tests. This helps make sure chatbot evaluations are more reliable.
task-oriented conversational agentsbenchmarkslarge language modelsbenchmark evaluationpolicy coveragebenchmark consistencysynthetic benchmarkshuman annotations
Authors
Noam Koren, Roy Bar-Haim, Abigail Goldsteen
Abstract
Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed. Poor benchmarks may contain inconsistent tasks, simplistic scenarios, or limited policy coverage, leading to unreliable evaluations. We introduce a reference-free framework that uses LLM judges to assess benchmark consistency, complexity, and policy coverage, while providing actionable diagnostics of weaknesses. We validate the framework by demonstrating agreement with independent human annotations and by evaluating benchmarks generated by LLMs of varying capabilities, as well as benchmarks subjected to controlled quality-degrading perturbations. Across domains and judge models, the proposed metrics consistently distinguish between benchmark quality levels. We further demonstrate the framework's applicability to manually curated benchmarks. Our framework offers a practical approach for evaluating synthetic and manually curated conversational-agent benchmarks.