Online decentralized learning finds equilibrium in multi-agent stochastic games
Fully Online Decentralized Learning in Stochastic Games with Unknown Independent Chains
Machine LearningComputer Science and Game TheoryMultiagent Systems
Summary
The paper deals with games where many players make decisions based on their own changing situations and unknown rules. The authors designed a method where each player learns how to act just by using their own experience, without needing to know everything going on with other players. This learning method aims to find a stable strategy where no player benefits from changing their plan alone. Their approach works well even for complex games with many players, avoiding huge increases in difficulty. Under certain conditions, their method can guarantee gradually better decisions over time and eventually approach stable strategies.
What this means in practice
- •For multi-agent system developers: Build scalable learning strategies where agents independently learn near-equilibrium behaviors without centralized control or full state knowledge.
- •For robotics teams: Develop algorithms enabling robots to adaptively coordinate strategies in uncertain environments using only local observations and rewards.
Authors
S. Rasoul Etesami
Abstract
We consider stochastic games with independent controlled chains and unknown transition kernels, where players observe only their local states and realized payoffs. We develop a fully online, decentralized, and uncoordinated mirror-descent algorithm that operates in the dual space of occupancy measures for approximating stationary Nash equilibrium (NE) policies. The algorithm uses a single transition/reward sample at every primitive time step, relies only on local information, and requires neither coverage of the joint state space nor synchronized episodes. Under uniform-ergodicity and finite-coverage assumptions, we show that, with high probability, the time-averaged fixed-comparator regret decays at the canonical $O(T^{-1/2})$ rate, up to logarithmic factors and polynomial dependence on the game parameters. In particular, the complexity depends on the cover times of the individual local state spaces rather than the product state space, avoiding exponential dependence on the number of players and the sizes of the joint state and action spaces. The resulting finite-time regret bound further yields an approximate coarse-correlated-equilibrium guarantee, which is natural for arbitrary reward functions since computing a stationary $ε$-NE is PPAD-hard in this setting. Under an additional global variational-stability condition, we show that the same fully online algorithm converges asymptotically in the last iterate to a stationary $ε$-NE. Our results provide a fully online and scalable learning framework for stochastic games with unknown independent chains. The algorithm can also be viewed as a primal-dual framework for Markov games that exploits the independence and local structure of the players' controlled transition chains.