Human-residual steering improves alignment of llm judgments with humans
Human-Residual Steering: Aligning Frozen LLM Judges with Human Judgment Distributions
Computation and Language
Summary
Large language models (LLMs) are often used to judge or rate things automatically, but their judgments don't always match the variety of opinions given by real people. The researchers developed Human-Residual Steering (HRS), a way to adjust an LLM’s internal processes to better reflect the spread of human judgments without changing the main model itself. HRS works by learning small corrections inside the model to capture different individual human ratings. This method performed better than previous approaches on many test cases and helped the LLM judges more closely match human judgment patterns.
What this means in practice
- •For machine learning engineers: Improve automated rating systems by better aligning LLM judges to reflect diverse human opinions using low-rank hidden state corrections.
- •For quality assurance teams: Increase reliability of AI-powered evaluation tools by reducing discrepancies between automated and human judgment distributions without retraining entire models.
Authors
Chen Lyu, Phong Do, Gabriele Pergola
Abstract
Large language models (LLMs) are widely used as automated judges, yet most alignment methods target aggregated human judgments and overlook variation among individual annotators. Existing approaches to matching the full distribution of human ratings mainly rely on output-level calibration or parameter fine-tuning. We propose Human-Residual Steering (HRS), to our knowledge the first method to align frozen LLM judges with human judgment distributions through learned hidden-state interventions. An oracle analysis shows that low-rank corrections at a single layer remove most of the discrepancy between judge and human distributions. HRS learns to predict such corrections from the judge's hidden representation and rating logits, training on individual annotations with proper scoring rules for ordinal and nominal judgments and optimising through the frozen downstream layers. Across 14 benchmarks and four judges from three model families, HRS outperforms all evaluated frozen-weight baselines. With Qwen2.5-7B, it achieves the lowest Jensen-Shannon distance on 13 of 14 benchmarks, reducing the mean from 0.565 to 0.352 versus 0.377 for the strongest calibrator. HRS also raises mean Spearman correlation with aggregate human scores from 0.449 to 0.579, improves population-level estimation under limited annotation budgets, and trains about 88 times fewer parameters than distributional fine-tuning at a modest cost in alignment.