Reinforcement learning wins more RTS games by mixing strategies
Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Games
Artificial Intelligence
Summary
Playing real-time strategy games well with AI is hard because opponents can act in unexpected ways. The authors created a system that separates deciding what overall game strategy to use from how individual units move and fight. This system uses a smart method to pick strategies based on what the opponent is doing, not just who they are. It played better than a baseline AI against several strong opponents, showing more wins on test maps.
What this means in practice
- •For game ai developers: Improve game AI by separating high-level strategy decisions from low-level unit control to adapt better against diverse opponents.
- •For robotics control teams: Use bandit-based strategy selection to switch between task commands while reusing stable execution policies under uncertainty.
Authors
Nick Leenders, Roy Lindelauf, Joost van Oijen, Boris Cule
Abstract
Deep reinforcement learning agents reach strong performance in real-time strategy games but can be brittle against opponents outside their training distribution. Separating strategic command selection from learned unit control allows different strategies to be selected for different opponents while reusing the same execution policy. This requires an executor that can follow different commands and measurable criteria for assessing whether it does so. We introduce a constrained command-conditioned Proximal Policy Optimization (PPO) policy, the executor, for MicroRTS, a real-time strategy environment. Discrete commands specify strategic objectives and behavioral requirements for economy, army composition, military posture, and worker policy over multiple environment steps; the executor determines the unit-level actions used to fulfill them. A Thompson-sampling bandit acts as the strategist, selecting command tuples from an estimate of the opponent's strategy built from in-game observations rather than opponent identity. In a controlled comparison with a flat PPO baseline trained with the same architecture, budget, curriculum and self-play league, the strategist-executor system wins significantly more often against three of the four strongest opponents on a training map, including the two strongest held-out ones (0.55 to 0.97 and 0.01 to 0.34), with no significant difference against the others.