Conditional Evaluation of Language Models with Cheap Auxiliary Signals
2026-08-17 • Machine Learning
Machine Learning
AI summaryⓘ
The authors present LACE, a new method to better understand where language models do well or poorly on specific types of tasks, without needing a lot of expensive human-labeled data. They use cheaper, imperfect signals like model confidence or judge scores alongside a smaller labeled set to get unbiased and more efficient evaluations. Their method mathematically ensures accurate estimates within groups of items and adjusts optimally based on the available signals. They tested LACE on several popular benchmarks, showing how it helps identify detailed model performance patterns.
conditional performancelanguage modelssemi-supervised estimationcontrol variatelocal centeringefficiencybias correctionperformance profilespaired comparisonsbenchmark evaluation
Authors
Zhi Zhang, Lingfeng Lyu, Yue Kang, Doudou Zhou
Abstract
Aggregate accuracy hides where models succeed and fail. Estimating conditional performance profiles from gold labels alone is expensive, while cheap auxiliary signals such as LLM-judge scores, pairwise comparisons, confidence scores, and judge-disagreement features can be collected for every benchmark item but are often biased or miscalibrated. We propose LACE (Local Augmented Control-Variate Evaluation), a semi-supervised estimator for conditional LLM evaluation. The key step is local centering: after subtracting the conditional mean of a cheap signal within the target profile region, any linear augmentation has zero conditional mean and therefore cannot change the estimand. The augmentation coefficient is used only for efficiency, and a local ridge control variate combines a gold-label residual mean from the labeled subset with a cheap-signal mean from the full item pool. We prove calibration-free identification, unbiasedness for grouped profiles, local oracle optimality within centered linear augmentations, and first-order adaptivity to the estimated coefficient. The resulting gain formula is governed by a population local $R^2$, which characterizes how the efficiency attainable from the cheap signals varies across profile values. We also derive corresponding estimators for direct paired model gaps and deployment-weighted scores. We empirically evaluate the primary performance-profile estimator on MATH-500, ScienceQA, MMLU, WinoGrande, HellaSwag, TruthfulQA, GSM8K, and ARC.