New method improves model tuning for off-policy evaluation tasks
Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation
Machine Learning
Summary
Off-policy evaluation helps computers learn from past experiences without trying new actions, but tuning the right models for this is hard because we can't directly check error on unused data. The authors introduce a new way called pairwise Bellman validation that compares pairs of models without extra complex steps and uses only simple calculations based on held-out data. Their method gives math-backed guarantees that the chosen model is close to the best possible, even if no perfect model exists. They test their approach on simulated tasks and find it works well compared to previous methods.
What this means in practice
- •For machine learning engineers: Tune off-policy reinforcement learning models more reliably without extra validation models or complex optimizations.
- •For autonomous systems developers: Improve evaluation of learned policies from logged data without deploying risky experiments by better selecting Q-functions and occupancy ratios.
Authors
Lars van der Laan, Vasilis Syrgkanis, Nathan Kallus
Abstract
Model selection and hyperparameter tuning pose a central challenge in off-policy evaluation (OPE) because errors of fitted Q-functions and occupancy-ratio estimates cannot be directly evaluated from held-out transitions. Bellman-residual and minimax validation methods often require auxiliary regressions or optimization over separate critic classes, adding computational and tuning costs. We introduce pairwise Bellman validation (PBV), a tuning-free minimax procedure for selecting among finite classes of fitted Q-functions and occupancy-ratio estimators using held-out transitions. PBV constructs critics from pairwise candidate differences, requiring only finitely many empirical moment calculations without auxiliary validation models or separate critic optimization. We establish finite-sample oracle inequalities relative to the best available candidate, even under candidate-class misspecification. Our key insight is that Q-function errors and log ratios between candidate and true occupancy ratios provide oracle critics whose Bellman and adjoint Bellman moments control estimation error. Pairwise candidate differences approximate these unknown critics, while penalties account for comparator approximation error. For Q-function selection, we first develop unweighted PBV, which evaluates pairwise Bellman moments under the offline distribution and admits oracle guarantees under a pairwise coercivity condition relating the offline distribution to the target-policy dynamics. We then develop target-aligned PBV (TA-PBV), which uses marginalized importance weighting to provide error control without pairwise coercivity. For occupancy-ratio selection, we develop adjoint PBV (A-PBV), which evaluates adjoint Bellman moments against log-ratio critics. On Hopper and Deep OPE, unweighted PBV performs competitively, while target-occupancy weighting can improve performance in settings with controlled overlap.