Improving offline policy evaluation using mixed step errors and adaptive critics
Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
Machine Learning
Summary
Estimating how good a decision-making strategy is using data collected from a different strategy is tricky. The authors propose combining simple one-step and more complex two-step error measures to strike a better balance between bias and variance in predictions. They also create a flexible way to represent the critic function that adapts to important features of the environment dynamics. This helps focus on relevant information when estimating the value of the strategy. Tests show this approach can improve the accuracy of estimates in challenging scenarios.
What this means in practice
- •For ai algorithm developers: Estimate policy values more accurately when training reinforcement learning agents from fixed offline datasets.
- •For robotics engineers: Improve evaluation of robot control policies using pre-collected task data without requiring new physical trials.
Authors
Amitakshar Biswas, Yuhan Li, Ruoqing Zhu
Abstract
Evaluating a target policy using data generated by a different behavior policy remains a fundamental challenge in reinforcement learning. While most existing work relies on the standard one-step Bellman residual, we consider a convex combination of one-step and two-step residuals with a fixed mixing weight. In ideal settings, this mixed Bellman formulation can provide a natural bias--variance trade-off between approximation error under a restricted value-function class and the increased variance arising from multi-step importance weighting. To solve this mixed residual optimization, we adopt a minimax formulation involving a critic function. Unlike standard approaches that rely on a fixed functional class, we construct a data-dependent critic representation using predicted future feature directions which effectively induces a kernel adapted to the underlying transition dynamics. This allows the critic to focus on directions that are most relevant for the estimated Bellman error. To control overfitting, we use sample splitting to construct the critic and estimate the value function on separate data subsets. Simulation studies and MetaWorld tasks illustrate the effect of the mixing parameter and show that intermediate residual combinations can improve value estimation in challenging settings.