Optimal transport improves safe evaluation of language models offline
OTROPE: Optimal Transport-based Robust Off-policy Evaluation for Large Language Models
Artificial IntelligenceComputation and Language
Summary
Evaluating new language AI models can be hard because it’s risky and costly to test them live, and there’s limited human feedback available. The authors propose a method called OTROPE that uses math from optimal transport to compare data from an old model to new model responses without needing detailed probabilities. This approach helps make better use of scarce human-labeled examples and can work even when models behave differently. Their experiments show it reliably outperforms older methods by combining corrected human data with predictions from other AI models.
What this means in practice
- •For ai product developers: Evaluate new language AI models safely offline using limited human feedback without needing direct model probabilities.
- •For nlp engineering teams: Improve model evaluation accuracy under changing data by aligning older labeled responses to newer model outputs in semantic space.
Authors
Liner Xiang, Wenbo Zhang, Hengrui Cai
Abstract
Reliable evaluation of large language models (LLMs) is essential for their development and deployment, yet is often costly, risky, and difficult to perform safely online. We study off-policy evaluation for LLMs, where limited human-labeled data from a behavior model are used to evaluate a newer target LLM. This setting is challenging because labels are scarce, behavior--target distribution shift is common, and response likelihoods are often unavailable for black-box LLMs. We propose the Optimal Transport-based Robust Off-Policy Evaluation (OTROPE), a likelihood-free evaluation that performs distributional correction in a semantic space via optimal transport to align labeled behavior-policy samples with unlabeled target-policy samples. OTROPE combines corrected human-labeled residuals with proxy predictors, yielding a doubly robust-style evaluation without behavior-policy modeling or density-ratio estimation. We theoretically characterize why baseline evaluators fail under LLM distribution shift, and establish consistency and convergence rates for OTROPE when either the reweighted behavior distribution or the proxy predictor converges. Experiments on synthetic and real LLM evaluation tasks show that OTROPE consistently outperforms baselines while enabling ensembles of weaker LLM evaluators to approach and sometimes surpass stronger evaluators. Code is available at https://github.com/LinerXiang/OTROPE.