Multi-objective reinforcement learning improves agent teamwork and trade-offs
Preference-Guided Cooperation in Multi-Objective Multi-Agent Reinforcement Learning
Multiagent SystemsMachine Learning
Summary
When many smart agents work together to solve a problem with several goals, figuring out how they should cooperate and balance these goals is really hard. The authors explore whether the fact that there are multiple goals can help agents cooperate better rather than just making the problem more complicated. They design a new method called MIRL that helps each agent learn which goals to focus on while still working well as a team. They test their idea in a simulated environment and show it helps the group find better solutions than previous approaches.
What this means in practice
- •For robotics teams: Coordinate robots tasked with multiple goals by improving their joint decision-making on which tasks to prioritize.
- •For traffic control operators: Optimize traffic signal controllers balancing various objectives like flow efficiency and safety using improved multi-agent cooperation.
Authors
Qiyue Xia, J. Michael Herrmann
Abstract
Multi-objective multi-agent reinforcement learning (MOMARL) addresses agent coordination and objective trade-offs simultaneously while optimising overall system performance. Existing work tends to treat multi-agent cooperation and multi-objective optimisation as largely independent components of a learning problem, which amounts to an exorbitant complexity except in trivial cases. We study here whether the structure provided by multiple objectives can itself be exploited to facilitate cooperation, rather than being treated as an additional optimisation challenge. We formulate MOMARL at both the team and individual levels and introduce objective-wise counterfactual credit assignment to evaluate each agent's contribution across the team objectives. We propose Meta-Gradient Informed Multi-Objective Multi-Agent Reinforcement Learning (MIRL), with static (MIRL-S) and dynamic (MIRL-D) variants. By incorporating meta-gradient information, MIRL enables agents to learn how to prioritise objectives individually while collectively optimising the system-level objective, allowing the multi-objective structure to directly guide cooperation. Experiments on a multi-objective multiple particle environment show that our methods improve Pareto-front quality over baselines, with the dynamic variant achieving the strongest performance.