Reinforcement learning achieves stable low regret with fine time steps
Frequency-Uniform Log-Regret Reinforcement Learning for Diffusions
Machine Learning
Summary
When computers learn to control things that change continuously over time, breaking time into smaller steps usually makes learning harder. The authors study how to keep learning efficient even as these time steps get very small, using a method based on approximating continuous changes. They show that under certain mathematical conditions, the learning process keeps improving with very little regret, meaning it quickly gets close to the best possible control strategy. This work bridges continuous-time models and discrete learning algorithms with guaranteed performance.
What this means in practice
- •For robotics engineers: Design control algorithms that maintain performance guarantees when using fine timing in continuous control settings.
- •For quantitative finance teams: Implement model-based learning methods for portfolio or risk control that remain effective despite high-frequency discretization.
Authors
Jose Blanchet, Lutong Hao, Renyuan Xu
Abstract
High-frequency control problems are often studied through time discretization, but refining the timestep $δ$ increases the discrete horizon $H=T/δ$ and can make standard finite-horizon regret guarantees deteriorate even when the physical horizon $T$ is fixed. We study episodic value-based reinforcement learning for Euler discretizations of controlled diffusions. The performance is measured as the cumulative regret over $K$ episodes relative to the optimal policy of the corresponding Euler model. Under stability and regularity conditions on value function gradients, Bellman residual derivatives, and fitted critics, we prove an $\mathcal{O}(\log K)$ regret bound with constants uniform in $δ$. For dynamics with drift linear in unknown parameters and unknown constant diffusion covariance, we verify localized versions of these conditions for a constrained, model-based controlled-diffusion fitted-$Q$ algorithm (CD-FQ) and obtain high-probability $\mathcal{O}(\log K\log\log K)$ regret for fixed and finite $T$, again with constants uniform in $δ$.