New framework improves learning in open multi-agent reinforcement systems
An Axiomatic Framework for Reinforcement Learning in Open Multi-Agent Systems
Machine LearningArtificial IntelligenceMultiagent Systems
Summary
Teaching multiple agents to work together can be tricky when only some of them can be controlled. The authors studied this problem by combining ideas from teamwork games and fair value sharing in groups. They created a method that treats these ideas as rules for learning, improving how agents learn from each other. Their approach works better than some existing methods, especially when dealing with new or unknown agents.
What this means in practice
- •For smart grid operators: Enhance coordination algorithms for partially controllable devices in smart grids using axiomatic value learning to improve efficiency and adaptability.
- •For robotics development teams: Develop robot teams that learn to cooperate better with unknown robot types in changing environments through structured value-sharing methods.
Authors
Jianhong Wang, Yang Li, Samuel Kaski, Jonathan Lawry
Abstract
Open multi-agent systems, where only a subset of agents are controllable, are central to real-world domains such as smart grids, yet lack a principled foundation for reinforcement learning algorithm design. We study this setting through n-agent ad hoc teamwork (NAHT), a recent framework for learning in open multi-agent environments. We propose an axiomatic framework that derives value function learning from the Shapley axioms (Efficiency, Symmetry and Linearity), treating them as structural constraints rather than heuristic design choices. This formulation builds on a novel NAHT decomposition, which characterises the fundamental difference between open and fully controllable multi-agent systems, and establishes a cooperative game-theoretic foundation for learning. We further show that enforcing these axioms on learning individual value functions recovers the Shapley value for Dec-POMDPs, providing a principled bridge between reinforcement learning and classical cooperative game theory. Building on this framework, we introduce Shapley Machines and Banzhaf Machines, two generic algorithmic templates that can be instantiated with standard reinforcement learning methods. When applied to strong base algorithms such as IPPO and POAM, our methods consistently improve learning efficiency and generalisation to unseen agent types and conventions across NAHT benchmark tasks. The code of experiments is published on https://github.com/hsvgbkhgbv/axiomatic-rl-naht.