Screening AI judges shows promise for dental hygiene coaching models
Screening LLM Judges Without Ground Truth: A Preliminary Evaluation of an AI Dental Hygiene Coach
Artificial Intelligence
Summary
Evaluating AI models by experts is slow and costly. The authors tested a step-by-step method to check if language model judges can reliably evaluate AI dental coaching models without needing expert scores first. They found that some AI judges gave consistent results similar to hosted ones, and the pipeline helped spot problems hidden in raw scores. However, these findings are early and no expert dentists reviewed the AI feedback yet.
What this means in practice
- •For ai product developers: Use the screening pipeline to quickly vet AI judges before engaging experts for feedback on coaching models.
- •For healthcare ai teams: Detect unreliable or biased AI judge outputs early in development of clinical coaching assistants.
Tested on simulated data.
Authors
William DeShong, Barath Kannan, Auberon Palma, Brenda Bradshaw, James F. Leathrum,, Yuzhong Shen
Abstract
Evaluating a fine-tuned coaching model with expert graders is slow and expensive, and an LLM judge cannot be trusted without ground truth. We present a staged pipeline that screens candidate LLM judges and student models before expert review, using explicit decision rules at each stage: judge output audit, similarity with hosted judges, bias checks, and decision robustness. We demonstrate it on an AI dental hygiene coach. The case study uses synthetic, clinically unvalidated training and reference data to demonstrate the operation of the pipeline, rather than to establish the clinical performance of the resulting models. Two fine-tuned small language models (Granite-4.1-8B and Qwen3-8B), each trained at three data sizes, were scored on seven metrics by five judges (three local, two hosted). Two local judges, Qwen3.8-27B and Gemma-4-31B-it, were not distinguishable from the agreement between the hosted judges on six of seven metrics by rank correlation, and Qwen3.8-27B was the most similar to the hosted judges overall. No judge met our criterion for favoring its own model family or training-data source. The student comparison did not separate the two models on any metric under the pipeline's rule, so the choice of Granite-4.1-8B, which led in 29 of 35 judge-by-metric cells, is a lean that depends partly on truncated Qwen3-8B outputs. The pipeline also surfaced problems that raw scores hid, including the truncated outputs and hallucination flags from one hosted judge that no other judge corroborated. No dental hygiene expert has graded these responses, so the findings are preliminary and the pipeline's final stage, anchoring to expert judgement, is proposed but not performed.