Orchestrating reasoned policies improves play against unknown opponents in dynamic games
Orchestrating Level-$K$ Policies Against Unknown Opponents in Partially-Observable Dynamic Games
Multiagent SystemsRobotics
Summary
When playing games where you don't know how smart your opponent is or what they will do, it helps to have a set of plans tailored to different opponent types. The authors show that instead of just guessing your opponent's type and sticking with one plan, it's better to dynamically switch between plans to get the best results based on new information during the game. They test this idea in a chase game and find that training a system to pick the best plan through trial and error works better than classification methods alone. This suggests that having a library of opponent-aware plans is useful if managed smartly, not just used directly.
What this means in practice
- •For game ai developers: Manage a set of opponent-aware strategies dynamically to improve AI performance against unknown player types in partially observable games.
- •For security simulation teams: Enhance multi-agent simulation systems by orchestrating prebuilt policies to adapt to unknown adversary behaviors in complex, uncertain environments.
Authors
Addison Kalanther, Sanika Bharvirkar, Daniel Bostwick, Chinmay Maheshwari, Shankar Sastry
Abstract
Level-$K$ reasoning generates a hierarchy of policies specialized to opponents with different reasoning levels. When an opponent's level is unknown, a common deployment rule estimates that level and selects the corresponding response. In a dynamic, partially-observed game, this selection is repeated, with each choice shaping subsequent states and observations. The response associated with the most likely opponent level need not maximize expected return from the current history. We formulate this deployment problem as dynamic orchestration of a fixed, pretrained policy library in a partially observable Markov game. We compare classification-based orchestrators (CBOs) trained using offline data or on-policy data aggregation with a reinforcement-learning-based orchestrator (RLBO) trained to maximize expected discounted return. In pursuit-evasion experiments, on-policy training improves classification and return, yet RLBO achieves higher return than the on-policy and offline CBOs. Given a pretrained library, RLBO also reaches performance comparable to a policy trained directly over the pursuers' action space with fewer training timesteps. These findings support treating a hierarchy of level-$K$ policies as a resource for orchestration, not a prescription for deployment.