Reinforcement learning improves constrained stochastic control in continuous time
Reinforcement learning for constrained stochastic linear--quadratic control in continuous time
Machine Learning
Summary
Controlling systems with uncertain behavior and limited control options is hard, especially when trying to plan for the best long-term outcome. The authors study a simple but important case where the system’s state changes in continuous time and controls must stay positive. They develop mathematical tools to find optimal strategies that balance exploring new actions and minimizing costs. Their approach also learns good control strategies without knowing the system beforehand. This work helps understand and solve control problems where randomness and restrictions matter.
What this means in practice
- •For industrial control engineers: Design controllers for processes with uncertain inputs and positive-only control limits using learning methods that adapt without detailed system models.
- •For robotics software developers: Create reinforcement learning algorithms for robots operating in continuous time where control actions must be nonnegative and system dynamics are initially unknown.
Authors
Weiping Wu, Qitong Pan, Xun Li, Peng Wu
Abstract
We study continuous-time stochastic linear--quadratic control with a scalar state and nonnegative control constraints. With quadratic running costs, randomized exploration can change the optimal mean feedback. We introduce an auxiliary problem without entropy in which the conditional policy mean is nonnegative and the covariance is prescribed in proportion to the squared state. Moment reduction and state separation yield two regional Riccati equations and characterize the finite-horizon value function and optimal conditional mean. We give a sufficient control--cost compatibility condition under which this mean coincides with the optimal feedback of the original constrained problem. For time-invariant coefficients, we characterize the undiscounted infinite-horizon optimum and establish mean square asymptotic stability under the stated assumptions. For unknown dynamics, we develop model-free actor--critic schemes for finite and infinite horizons. Numerical experiments compare the learned policies with their analytical targets and examine the effect of exploration on the optimal mean.