LLM judges improve reliability estimation using verdict-conditioned signals
VERA: Verdict-Conditioned Reliability for Adaptive LLM Judges
Machine Learning
Summary
Judging how reliable a large language model (LLM) is at giving correct answers can be tricky because usual confidence scores can be misleading. The authors introduce a new method called VERA that looks inside the model’s hidden layers to better tell when a judgment is likely right or wrong for each type of verdict it gives. They then use this method to guide retraining of the model, helping it learn from new feedback without forgetting old knowledge. This approach improved performance on multiple benchmarks and specific auditing tasks compared to previous adaptive methods.
What this means in practice
- •For llm system engineers: Improve model updating processes by integrating hidden-layer based reliability signals to balance learning new feedback while retaining prior behavior.
- •For compliance and audit teams: Enhance the accuracy of automated evaluation tools by using verdict-conditioned reliability to better detect errors in temporal auditing processes.
Authors
Qiushui Xu, Syamil Mohd Razak, Tao Yuan, Piotr Habas
Abstract
Accurately estimating judgment reliability is a central challenge in adapting LLM judges to newly verified feedback while preserving previously learned behavior. However, existing approaches often rely on output-level confidence, which can be overconfident and poorly aligned with judgment correctness. We propose VERA, a VErdict-conditioned Reliability Axis that estimates reliability from hidden activations by distinguishing correct from incorrect judgments within each predicted-verdict group. Using VERA as a control signal, we develop a VERA-guided periodic adaptation framework that integrates reliability-ranked corrective updates, reliability-residual replay, and periodic refresh of the reliability directions. After VERA-guided adaptation on Chatbot Arena, 8B- and 14B-parameter judges outperform the strongest baseline on each of four held-out public benchmarks, with relative gains of up to 23.01%. The framework also improves focal-class recall by up to 16.1% relative to the strongest adaptive baselines on a separate proprietary temporal auditing task.