Schrödinger Föllmer Actor Critic improves policy learning with diffusion methods
Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis
Machine Learning
Summary
Finding the best way to choose actions in complicated situations is hard, especially when actions can have many possible outcomes. The authors propose a new method that uses ideas from diffusion processes to improve how policies are updated in reinforcement learning, combining offline and online data. Their method estimates the advantage of actions and applies a mathematical correction that guides the policy improvement efficiently. They test their approach on simulation tasks and show it performs better than starting policies alone.
What this means in practice
- •For robotics engineers: Improve robot control policies by refining decision-making using diffusion-based policy updates that combine offline data with online fine-tuning.
- •For automated trading teams: Enhance trading algorithms by allowing more flexible action distributions and updated strategies using diffusion policy improvement under uncertainty.
Authors
Yuling Jiao, Lican Kang, Jerry Zhijian Yang, Jincheng Ying
Abstract
Diffusion policies represent multimodal action distributions, but an advantage-weighted update does not specify how to sample from the resulting target distribution. We propose Schrödinger--Föllmer Actor--Critic (SFAC), an offline-to-online reinforcement learning (RL) method for Kullback--Leibler (KL)-regularized policy improvement through conditional diffusion. A minimax Bellman critic estimates the advantage function, which defines an exponentially tilted target policy. A Doob $h$-transform expresses this update as a correction to the reference diffusion drift. We derive a posterior-mean representation of the correction and estimate it using paired self-normalized importance sampling (SNIS). Supervised regression on these drift targets updates the neural actor without critic action gradients. In the small-update regime, the KL-regularized update follows the natural policy-gradient direction, and the Doob correction represents the same local change in the space of diffusion drifts. Under suitable conditions, we derive finite-sample bounds that separate the effects of critic estimation, neural drift regression, finite-sample SNIS, diffusion discretization, and inherited actor error on expected average policy suboptimality. Synthetic experiments assess the accuracy of approximation to prescribed advantage-tilted targets and sensitivity to sampling budgets. On six offline-to-online continuous-control tasks, a reference-anchored implementation achieves higher final-window returns than those of its corresponding offline initialization.