Amortized off-policy evaluation lowers error for language models
Amortized Off-Policy Evaluation for LLMs
Machine Learning
Summary
Choosing the best language model to use can be tricky because testing new models directly on users can cause problems if the models are not good yet. The authors introduce a new method to estimate how well a new model will perform using past data from older models, even when the way performance is judged changes. Their method, called PFN-OPE, can quickly predict the quality of new language model responses without needing to learn from scratch each time. Tests showed this method is much more accurate than existing approaches.
What this means in practice
- •For machine learning engineers: Estimate new language model performance offline faster and more accurately for safer deployment decisions.
- •For recommendation system teams: Evaluate updated recommendation models on historical user interaction data despite changing evaluation criteria without retraining.
Authors
Younwoo Choi, Leo Feng, Vincent Liu, Haanvid Lee
Abstract
Accurate evaluation is central to selecting which LLM to deploy, yet testing a candidate on live traffic exposes real users to an unvetted model. Teams therefore evaluate candidates offline, on data produced by already-deployed models. This is off-policy evaluation (OPE), and it faces two distribution shifts: as a model is updated in post-training, its responses diverge from the logged ones (policy shift), and the reward definition under which it is judged changes with business requirements (reward shift). Classical OPE methods are ill-suited to this continual-deployment setting because they are defined per task and require fitting from scratch on every new logged dataset or reward definition. To address this, we propose PFN-OPE, a prior-data fitted network that amortizes OPE across a distribution of contextual-bandit tasks. We pretrain it once on tasks constructed from a pool of LLM responses scored by several reward functions, in which both shifts occur. At test-time it maps a logged dataset and one sampled target response per prompt to a value estimate in a single forward pass, with no per-task fitting. On HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies, PFN-OPE achieves 2.0 to 9.3 times lower error than the best baselines across all tested configurations in the reward-shifted settings.