Reference policies guide equilibrium choice in strategic self-play
Steering Equilibrium Selection in Regularized Self-Play via the Reference Policy
Artificial IntelligenceComputer Science and Game TheoryMachine Learning
Summary
In some strategic games, there can be many equally good ways to play, making it unclear which strategy a learning algorithm will choose. The authors study a method that nudges the learning process toward a specific desired strategy by tweaking a guiding reference policy. Their experiments confirm this approach works precisely when tested on small games where exact calculations are possible. They also identify limitations and conditions where steering the choice is harder or less reliable. This helps clarify how a guiding policy influences outcomes in algorithms that learn by playing against themselves.
What this means in practice
- •For game ai developers: Control which Nash equilibrium an AI converges to by anchoring the reference policy to steer learning toward desired strategies in zero-sum games.
- •For reinforcement learning engineers: Tune reference policies during training to ensure stable and interpretable policy convergence when applying entropy-regularized self-play methods.
Authors
Luis Leal
Abstract
Regularized self-play -- the family behind DeepNash's Stratego play -- drives a two-player zero-sum policy to a Nash equilibrium by best-responding to a slowly moving, entropy-regularized reference policy $ρ$. When the game has a polytope of value-equivalent equilibria, the regularizer silently breaks the tie: with a uniform reference it selects the maximum-entropy member, the I-projection of $ρ$ onto the Nash set. Can the reference be used to choose the equilibrium on purpose? On five exactly solvable games plus a 2-D polytope, with exact best responses and equivalence tests over independent seeds, anchoring the reference at a target member and refining steers self-play to that member with mean coordinate error 0.007 at median exploitability $5\times10^{-5}$, TOST-equivalent to the request within $\pm0.05$; the anchoring persists through refinement and follows the reference, not the initialization. Selection follows the reach-weighted I-projection (slope 0.969 [0.950, 0.987]). We report with equal emphasis where the story breaks: fixed off-manifold references cost 0.08-0.25 exploitability; stiff or flat families require a smaller mirror step, set by a pre-registered rule; boundary targets undershoot; curvature predicts where boundary saturation bites (rank correlation 0.90, p=0.037) while interior precision is curvature-independent. Table and MLP steering maps are equivalent within $\pm0.03$ at every target (30 seeds); matched control arms show attention's robust signature is excess seed variance, any systematic shift bounded at 0.018 and not significant. Against a best response the selection-robustness trade-off is degenerate: steering matters only against fixed, non-equilibrium opponents. The recipe -- anchor the reference at the desired member and refine -- reinterprets the KL anchor of RLHF-style RL as a selection knob, not only a stability leash.