Bellman Calibration for Marginalized Importance Weighting in Offline Reinforcement Learning
2026-08-25 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study a way to better estimate how well a target policy performs using data collected offline, focusing on correcting errors in existing methods that weigh state-action samples. They propose a new technique called isotonic Bellman calibration, which adjusts the estimated weights to better satisfy balance conditions without changing their ranking order. This method is simple, model-independent, and improves accuracy by fitting a nondecreasing transformation to the initial estimates. The authors provide theoretical guarantees showing their method closely matches the best possible correction and supports reliable policy evaluation.
Marginalized importance weightingOccupancy ratioBellman equationOffline policy evaluationFitted fixed-point estimatorIsotonic regressionCalibrationKL divergencePolicy value estimationMonotone transformation
Authors
Lars van der Laan, Nathan Kallus
Abstract
Marginalized importance weighting evaluates a target policy by reweighting offline state-action samples with its discounted occupancy ratio, characterized by an adjoint Bellman equation. Existing minimax, primal-dual, and fitted fixed-point estimators can leave residual occupancy-balance violations because of function-class approximation, regularization, or incomplete optimization. These violations are difficult to diagnose and reduce because the objectives generally lack a direct supervised validation loss for hyperparameter tuning, model selection, and early stopping. We introduce isotonic Bellman calibration, a one-dimensional, model-agnostic post-processing method that reduces these violations while preserving the ranking information in any initial occupancy-ratio estimate. The method corrects the estimate's scale and shape by applying fitted occupancy-ratio evaluation (FORE) over a one-dimensional class of nondecreasing transformations. We characterize Bellman calibration as a conditional fixed-point property equivalent to occupancy-balance against every test function of the calibrated ratio. More generally, we derive a calibration-refinement bound showing that any fitted ratio with small calibration error performs nearly as well as the best post-processing based on its fitted values. For isotonic Bellman calibration, we establish finite-sample calibration guarantees and a KL oracle inequality relative to the best monotone transformation of the initial estimate. Consequently, isotonic Bellman calibration achieves small calibration error and KL risk within statistical error of the best monotone correction, with guarantees for downstream target-occupancy functionals, including policy-value estimation.