Diffnash finds better strategies in high-dimensional continuous games
DiffNash: Finding Nash Equilibrium in High-Dimensional Continuous Action Spaces with Diffusion Policies
Machine LearningArtificial IntelligenceComputer Science and Game TheoryMultiagent Systems
Summary
Finding the best strategies for games where players can choose from many complex options is hard, especially when traditional methods struggle to represent all possible moves. The authors created DiffNash, which uses a special kind of AI model called diffusion policies to handle complex and changing strategy choices. This approach improves how quickly and accurately the best responses are learned by reusing past information cleverly. They tested DiffNash on multiplayer strategic games and showed it finds stronger strategies than previous methods with less tuning.
What this means in practice
- •For ai game developers: Compute stronger strategies for continuous-action multiplayer games to improve AI performance in simulated competitive environments.
- •For security simulation teams: Model defense strategies in scenarios like protecting forests by finding equilibria among competing agents with many possible actions.
Authors
Haichuan Wang, Anagha Satish, Jiale Zheng, Saisneha Ghatti, Lingkai Kong, Kai Wang, Milind Tambe
Abstract
Computing equilibria in high-dimensional continuous games requires learning and representing rich, multi-modal strategy distributions, something conventional neural policies struggle to achieve. Diffusion models offer a natural solution, but standard diffusion training requires direct samples from a fixed target distribution, which are generally unavailable under iterative game-solving updates. We propose a diffusion-based framework for equilibrium computation that enables diffusion-based variants of Online Mirror Descent (OMD) and Fictitious Play (FP). Our approach derives a reweighted score-matching objective that trains diffusion policies without requiring direct samples from each updated policy. Our main method, DiffNash, combines the fast convergence of OMD with the sample efficiency of FP by reusing past actions through a replay buffer. To correct the resulting distribution shift from sample reuse, we derive an unbiased estimator of the difference between current and past policies' Evidence Lower Bounds, whose exponential provides a tractable proxy for the corresponding importance weight without evaluating diffusion likelihoods. We evaluate our approach on forest protection games and continuous Colonel Blotto games, reducing the Nash gap over the strongest baseline by 35.6% and 24.7%, respectively, with minimal hyperparameter tuning.