NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games
2026-09-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study how to improve learning in competitive games where players have hidden information. They argue that learning a central model of the game environment, rather than separate models for each player, is necessary to handle changing opponent behavior. To do this, they introduce NashDreamer, a method that uses a shared recurrent model to predict game dynamics and player observations, enabling efficient learning towards optimal strategies. Their experiments show NashDreamer learns faster than methods without such models. They also identify a potential issue with related algorithms collapsing under randomness, which remains unsolved.
Model-Based Reinforcement LearningImperfect Information GamesNash EquilibriumMulti-Agent SystemsOpponent Non-StationarityRecurrent State-Space ModelPolicy GradientSample EfficiencyPosterior Collapse
Authors
Tomáš Holeček, Viliam Lisý
Abstract
Model-based reinforcement learning (MBRL) has achieved remarkable results in single-agent domains, yet its extension to competitive imperfect information games (IIGs) remains underexplored. In multi-agent settings, opponent-induced non-stationarity complicates the learning process, and decentralized model learning faces severe identifiability barriers, which we argue make centralized model learning a mathematical necessity. Building on this analysis, we propose NashDreamer, a principled MBRL framework for two-player zero-sum IIGs. It introduces a centralized Multi-Agent Recurrent State-Space Model (MARSSM) that decouples environment dynamics from the effect of players' strategies on their individual observations. NashDreamer is designed to use arbitrary policy gradient algorithms and inherits their convergence guarantees towards Nash equilibria under an idealized model. Empirical evaluations across four benchmark games demonstrate that NashDreamer substantially improves sample efficiency over model-free baselines early in the training. Finally, we theoretically analyze the architecture's optimization landscape, identifying the vulnerability of the Dreamer family of algorithms to posterior collapse in stochastic environments. We leave it as an open challenge.