Mean field games with entropy regularization enable reliable average reward learning
Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning
Computer Science and Game TheoryMachine Learning
Summary
Some problems involve many decision makers whose actions affect each other, like drivers choosing routes. This paper looks at a mathematical model for such situations that includes a way to encourage exploring different choices, called entropy regularization. The authors show conditions where the model's calculations reliably settle to a unique solution, and they offer a method to learn these solutions from experience without knowing all details upfront. They demonstrate their approach with a traffic routing example where congestion matters.
What this means in practice
- •For traffic management teams: Improve routing strategies that adapt to congestion by learning equilibria without full system knowledge.
- •For network operators: Design adaptive resource allocation schemes that balance load efficiently using model-free equilibrium learning.
Tested on simulated data.
Authors
Gunes Akbas, Naci Saldi, Berkay Anahtarci, Can Deha Kariksiz
Abstract
We study stationary discrete-time mean-field games with finite state and action spaces under the average-reward criterion, with an entropy-regularized objective and population-dependent dynamics and rewards. Under a minorization condition, the soft Q-operator is a span-seminorm contraction for the full transition kernel, so relative value iteration does not require the minorizing measure. When the Lipschitz constant of the kernel is below the minorization mass and the temperature exceeds an explicit threshold, the equilibrium fixed-point operator is a contraction: an equilibrium exists, its population distribution is unique, and a fixed-point iteration converges to it geometrically. We then learn the equilibrium model-free from a simulator, combining soft Q-learning at a fixed population distribution with an empirical transition kernel. We prove a finite-sample bound for soft Q-learning in the span seminorm, with temperature-independent constants, and derive convergence rates for the learned population distribution and policy. A congestion-aware routing game illustrates the results.