Continuous-time reinforcement learning from fixed discrete data with guaranteed policy quality
A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees
Machine Learning
Summary
This paper tackles the challenge of learning better decision-making strategies when the environment changes continuously but data is only available at set times. The authors present a method that learns directly from these fixed batches of data without needing to build a full model of the system. They show how their approach balances accuracy in approximating continuous changes with errors introduced by using limited data, providing formal guarantees on the quality of the learned strategies.
What this means in practice
- •For control system engineers: Develop policies for systems modeled by continuous-time stochastic processes using only previously collected finite discrete data without system identification.
- •For financial quantitative analysts: Improve trading strategies with reinforcement learning that accounts for continuous-time market dynamics while relying on historical discrete data.
Authors
Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu
Abstract
In this paper, we study continuous-time reinforcement learning, where the state follows an unknown stochastic differential equation but only a fixed off-policy batch of discrete-time transitions is available. Standard discrete-time Q-learning becomes problematic as the sampling interval shrinks, because differences between action values vanish. Continuous-time alternatives, such as Deep Advantage Updating and martingale q-learning, preserve the action signal but jointly learn a value function and an action-dependent critic. Building on the PhiBE-Q formulation (arXiv:2606.21925), we develop a model-free PhiBE policy iteration (PhiBE-PI) framework that uses a single state-action critic while preserving the continuous-time action-ranking scale. At each iteration, the framework learns the critic directly from sampled increments and uses it to improve the policy. The same batch is reused throughout, without explicitly estimating the drift or diffusion. We further develop an implementation under linear function approximation and establish a local end-to-end suboptimality guarantee for the learned policy. Our analysis quantifies how errors from time discretization, function approximation, and finite-sample policy evaluation propagate through policy iteration. It accounts for the dependence of successive policies on the reused data, so the guarantee does not require a fresh batch at each iteration. Moreover, we identify a bias-variance trade-off: smaller sampling intervals improve the approximation to the continuous-time problem but can increase statistical error at a fixed sample size. Thus, finer time discretization does not necessarily lead to better policy performance.