Summary
When multiple AI agents learn to cooperate, they often start from scratch and find ways to work together. This paper looks at what happens after cooperation is already learned—if continuing to train them might break that cooperation. The researchers find that letting certain information, called value gradients, update shared parts of the agents' decision systems can sometimes destabilize cooperation, depending on how the training is scaled. However, keeping critics (evaluators) around without letting those gradients flow, or removing critics entirely, tends to maintain cooperation better. This means that the way learning updates are routed internally matters a lot for keeping cooperation stable.
What this means in practice
- •For multi-agent system developers: Design training procedures that protect learned cooperation by controlling how value gradients update shared policy parts in collaborative agent teams.
- •For ai optimization engineers: Adjust reward scaling and optimizer settings to reduce risks of destabilizing cooperation during extended training of multi-agent reinforcement learning models.
Authors
Chaoyuan Hao, Wentao Yue, Tianyou Lai, Hongji Li, Jiayi Zhou, Qingyu Mao, Qilei Li
Abstract
Cooperative MARL is commonly evaluated through cooperation discovery from random initialization, leaving open whether continued optimization can destabilize learned cooperation. Actor-critic comparisons can also conflate critic presence with value gradients entering shared actor representations. We study cooperation maintenance, defined as the survival of a behaviorally verified cooperative policy under continued training. We formulate maintenance as a right-censored event-time problem and compare matched warm starts: X0 allows value loss gradients to update shared actor features, X1 retains the critic while blocking those gradients, and X5 removes the learned critic as a critic-free reference. This isolates direct value-gradient access while controlling initialization, critic computation, and evaluation. Positive reward scaling preserves strategic preferences and equilibria while perturbing learning dynamics. Gradient audits confirm the intended routing pathways, and frozen-policy torso perturbations probe whether route-induced updates align with local cooperation boundaries. In confirmatory MinEx and CleanUp-lite experiments, higher scales selectively increase maintenance sensitivity in X0; X1 remains near the censoring ceiling, and X5 has no confirmed events in the tested settings. In CleanUp-lite, route-by-scale displacement is associated with reduced local cooperation margins; MinEx shows a weaker, optimizer-dependent effect. These results identify a conditional, scale-sensitive maintenance risk associated with direct value-gradient routing rather than a universal failure of critics.