Learning speed differences help agents cooperate in repeated games
Multi-Agent Learning of Reactive Strategies in Iterated Games
Computer Science and Game Theory
Summary
When people or computer programs learn to work with others, they usually try to improve their own outcomes. The authors study how different learning speeds among agents can affect cooperation in repeated situations like the prisoner's dilemma. They find that if all agents learn at the same rate, they tend to stop cooperating, but if learning speeds vary, cooperation can emerge and last. This cooperation happens because the slower learners keep being forgiving while the others adapt quickly.
What this means in practice
- •For multiagent system developers: Design multiagent learning algorithms that use varied learning speeds to promote sustained cooperation in repeated interactions.
- •For online platform managers: Improve user interaction dynamics on platforms by simulating different learning rates to encourage cooperative behavior among participants.
Authors
James P. Wolfe, Joshua B. Plotkin
Abstract
Learning agents, whether humans or algorithms, adapt to one another rather than to a fixed environment. Selfish learning need not produce a good outcome for the group, especially when learners face a social dilemma. The iterated prisoner's dilemma (IPD) is the classical example, but it is usually studied in populations of agents who imitate successful strategies. Here, we instead study selfish gradient ascent learners on the two-dimensional space of reactive strategies. A reactive strategy is described by two continuous traits: reciprocity, the chance of cooperating after the co-player cooperates, and forgiveness, the chance of cooperating after the co-player defects. Each learner updates its strategy in the direction that most rapidly improves its payoff against all other learners, by an amount governed by its learning rate. We show that heterogeneity in learning rates can qualitatively change the long-term outcome: a population with uniform learning rates converges to always-defect (ALLD), whereas a population with variable learning rates approaches generous-tit-for-tat (GTFT), which produces nearly optimal collective payoffs. In simulations, every population passes through a transient state of minimal forgiveness, in which the distribution of reciprocity determines the outcome. Variable learning rates promote cooperation because slow learners preserve high reciprocity until the population reaches this state. We prove sufficient conditions on the distribution of reciprocity for convergence to ALLD or to tit-for-tat (TFT). Variable learning rates also allow cooperation to recover from perturbation, which is provably impossible under uniform learning. These results suggest that purely selfish and myopic learning can produce persistent cooperation provided the population has intrinsic variability in learning rates.