Model based bootstrap improves uncertainty estimates for offline policy evaluation
Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning
Machine Learning
Summary
Deciding if a new way for a computer to make decisions works well without testing it live is very important, especially when mistakes are costly. The authors introduce a new method that creates many fake examples based on a learned model to better understand how certain we can be about the computer's predicted performance. This method works well with different types of offline data and gives more reliable confidence measures than older approaches. It helps people know not just the expected result but also how much the result might vary, making decision-making safer.
What this means in practice
- •For automated decision system engineers: Evaluate new decision policies safely by estimating risks accurately without deploying them in real environments using improved uncertainty measures.
- •For healthcare analytics teams: Assess treatment strategies offline with better confidence intervals when patient data comes in fragmented or incomplete formats.
Authors
Weiwei Wang, Yuqiang Li, Xianyi Wu, Bingyi Jing
Abstract
Offline policy evaluation (OPE) is crucial in high-stakes reinforcement learning applications, where new policies must be assessed reliably before deployment. In such settings, point estimates alone are insufficient; principled uncertainty quantification, such as confidence intervals and variance estimates, is essential for safe and risk-aware decision-making. A comprehensive way to unify these tasks is to estimate the sampling distribution of the evaluation error. Existing approaches, however, often suffer from limited robustness, scalability, or finite-sample validity. In this paper, we propose a model-based bootstrap framework for uncertainty quantification of OPE in finite-horizon, time-inhomogeneous Markov decision processes (MDPs). Unlike classical bootstrap methods that rely on resampling complete episodes, the proposed method regenerates trajectories from an estimated MDP and can therefore accommodate a much broader range of offline data formats, including complete trajectories, transition-level observations, and trajectory fragments. This flexibility further improves finite-sample statistical efficiency. We establish bootstrap distributional consistency, asymptotically valid confidence intervals, and consistent variance estimation for the target policy value. Extensive simulations show that the proposed method accurately captures the sampling distribution of the OPE estimator, yielding tighter confidence intervals and more accurate variance estimates in most settings.