Optimal value inference improves offline reinforcement learning estimates
Optimal Value Inference for Reinforcement Learning
Machine Learning
Summary
Estimating the best possible outcomes in decision-making problems is hard, especially when you only have past data to learn from. The authors developed new ways to better guess these optimal values by adjusting techniques that blend math and machine learning. Their method reduces bias and works well even when the data comes from changing decision patterns. They tested their approach on simulated examples and real tasks like bike sharing and AI tool use, showing it can help make better-informed decisions from offline data.
What this means in practice
- •For urban operations teams: Improve bike repositioning decisions using better estimates of optimal action values derived from historical system behavior data.
- •For ai developers: Enhance AI agent tool-use strategies by inferring optimal values from prior interaction data under evolving policies.
Authors
Nan Lu, Ethan Lee, James M. Robins, David Simchi-Levi, Junwei Lu
Abstract
We study offline inference for the optimal value in reinforcement learning. Two new nuisances are derived as fixed points of a self-induced Bellman equation, in which we approximate the maximum Bellman operator by its softmax correspondence. We propose a debiased estimator through the Neyman orthogonality and establish its asymptotic normality under diverging horizons even when the behavior policy changes with time, as long as the nuisances have the statistical rates that can be achieved by many machine learning methods. We provide a concrete estimating procedure for these nuisances and show they can lead to valid inference. Synthetic experiments validate the numerical performance of our inference method, and we implement it in real-life decision-making problems, including bike repositioning and AI agentic tool use.