Fitted Occupancy-Ratio Evaluation without Bellman Completeness
2026-07-06 • Machine Learning
Machine Learning
AI summaryⓘ
The authors introduce a new method called FORE to estimate occupancy ratios in offline reinforcement learning, which helps adjust for changes in the data distribution. Unlike previous methods, FORE focuses on directly approximating the discounted occupancy ratio using a fixed-point approach and projections based on KL divergence. Their method requires less strict assumptions than prior work, only needing the occupancy ratio to be realistically representable. They provide theoretical guarantees showing their approach converges well and can be used to improve value estimation in several ways. Overall, their results simplify the requirements for accurate offline policy evaluation.
occupancy ratiooffline reinforcement learningoff-policy evaluationBellman equationKL divergencefixed-point methodvalue functiondensity ratioreinforcement learning theory
Authors
Lars van der Laan, Nathan Kallus
Abstract
Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class. We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion. At each iteration, FORE solves a single-level density-ratio objective on one-step-transition data, thereby projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence. Unlike analyses of fitted Q-evaluation, which typically require value-function realizability together with Bellman completeness or projected-operator stability, our central approximation condition is just realizability of the discounted occupancy ratio itself. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio by virtue of the adjoint Bellman operator being a KL-contraction. For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to log-ratio approximation error and a statistical error governed by the complexity of the ratio hypothesis class. The fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation that combines the fitted ratio with a fitted Q-function. Together, these results identify discounted occupancy-ratio realizability as a sufficient condition for offline policy evaluation without any completeness assumptions.