Stochastic control bridges two mathematical frameworks with unified conditions
Unified Optimality Conditions for Stochastic Optimal Control in the Rough Path and Itô Frameworks
Machine LearningRobotics
Summary
Controlling systems affected by randomness is tricky and can be studied with two main mathematical tools: Itô calculus and rough path theory, each giving different rules for finding the best control steps. The authors show these two sets of rules are related through a clear connection involving expected values. They develop a new way to describe control problems in the rough path framework without relying on complex equations used before. Finally, they link these two approaches in a unified way, making it easier to move between them and apply new methods for improving models and control systems.
What this means in practice
- •For machine learning engineers: Refine generative models using a method based on adjoint matching derived from the unified stochastic optimal control framework.
- •For control system designers: Use an indirect shooting method for solving certain feedback control problems that leverages the unified Pontryagin conditions.
Authors
Thomas Lew
Abstract
Stochastic differential equations (SDEs) can be studied via Itô calculus and rough path theory. For stochastic optimal control, these two frameworks give distinct Pontryagin Maximum Principle (PMP) optimality conditions with forward-backward SDEs (FBSDEs) or rough differential equations. We show that the adjoint equations of the Itô and rough PMPs are connected via the conditional expectation $p_t^{\text{Itô}}=\mathbb{E}[p_t^{\text{rough}} \mid \mathcal{F}_t]$, where $\mathcal{F}_t$ represents information available at time $t$. First, we derive a rough stochastic PMP for problems with adapted controls that does not use FBSDEs. Its proof extends the rough stochastic PMP over deterministic controls by considering stochastic needle variations. Second, we derive a unified PMP connecting the Itô and rough PMPs, using Itô-Stratonovich conversion formulas and duality identities between the forward tangent and backward adjoint SDEs. As a first application, we rederive the adjoint matching method for fine-tuning generative models. As a second application, we propose an indirect shooting method for a class of feedback problems. Overall, these results give a new conditional bridge connecting two popular frameworks for stochastic optimal control.