Transport maps improve policy gradients in population control models
Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control
Machine Learning
Summary
Controlling behavior in large populations is tricky because changes affect individual outcomes and the whole group together. The authors study a way to teach computers to make better decisions in such settings without knowing all details beforehand. Their method, called Transport REINFORCE, uses a clever technique to estimate how changing group behavior affects the goal, improving on older methods. They tested this on examples and found it works better than standard approaches.
What this means in practice
- •For reinforcement learning engineers: Improve policy gradient methods for controlling large groups when model details are unknown.
- •For autonomous vehicle fleet operators: Optimize distributed fleet behavior accounting for interactions across many vehicles without detailed system models.
Authors
Adonis Jamal, Samy Mekkaoui, Yadh Hafsi, Huyên Pham
Abstract
We develop a model-free policy gradient method for discrete-time mean-field control (MFC). In MFC, the policy affects the objective both through the controlled dynamics and through the population distribution. Standard REINFORCE estimators capture the first effect but not the second. We introduce Transport REINFORCE, a transport map-based approach that perturbs a suitable transformation of the population distribution to estimate this missing mean-field contribution. The method applies to both finite and continuous state spaces. In finite state spaces, we perturb the population distribution directly on the probability simplex through a convex combination of the current population weights and random weights. In continuous state spaces, we project the population distribution onto the manifold of Gaussian mixtures, and then randomize it via a transport map that ensures the perturbed law remains within this manifold. We prove consistency of the perturbed objective and gradient as the perturbation vanishes, and derive bias and mean-square error bounds for the resulting sample-based gradient estimator. Numerical experiments on several MFC benchmarks show that Transport REINFORCE improves over standard REINFORCE.