Residual score adjustment can reduce bias but risks lowering quality alignment
When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits
Computation and Language
Summary
Evaluation scores for AI language models can mistakenly focus on how things look rather than how good they truly are. The authors show that removing predictable format signals from scores can reduce this bias, but it doesn’t always improve the true quality measurement. Sometimes, adjusting scores can even harm how well they reflect the real quality. They propose a careful way to report these adjusted scores, showing what the changes reveal and what costs they incur.
What this means in practice
- •For machine learning engineers: Improve auditing methods for AI model outputs by identifying and adjusting for superficial format biases without compromising quality judgments.
- •For software quality assurance teams: Use residualized scoring diagnostics to better understand evaluation score limitations when comparing correct and buggy code submissions.
Authors
Daein Weon, Dongho Kang
Abstract
Evaluation scores used around LLM systems -- including reward models, rerankers, and LLM judges -- can track surface form instead of the quality they claim to measure. When presented with a terse correct solution and a commented buggy solution for the same MBPP problem, a public preference reward model selects the correct one no better than a coin flip (0.507). Subtracting the predictable surface component from such scores is increasingly common, but removal alone does not yield a more valid measurement: the removed component may carry construct-relevant signal, and residualization cannot tell which is which. Under designed interventions -- unit-test labels with comment-only edits -- residualization attenuates the reward model's format effects by about 0.12 on both correct and buggy code, while the correct-versus-buggy margins move by less than 0.01. In observational NLI and QA settings, we freeze a held-out replication before scoring and re-evaluate it using labels from disjoint annotators; this supports only a narrower conclusion: better agreement with the construct labels on a pre-declared slice where a surface-only predictor errs, not a repaired score. Full-population agreement falls in every observational setting with a reported positive slice gain, and within-question ranking falls in every such QA setting. When construct and surface features are entangled, residualization can decorrelate a score while degrading construct alignment, and, in a controlled model, configurations just as damaging to construct alignment pass every pre-adjustment check, so no committed gate is a guarantee. We assemble these distinctions into a reporting protocol whose outcomes, refusal included, state what an adjusted score may be claimed to show: an audit-time diagnostic reported beside the construct-alignment cost it incurs, never a replacement for the raw score.