Reinforcement learning method reduces hidden algorithm collusion in repeated games
Mitigating Retaliatory Algorithmic Collusion in Repeated Games
Machine LearningArtificial Intelligence
Summary
Sometimes, computer programs that learn how to act in ongoing situations can end up cooperating secretly to get better rewards, even though they are not supposed to work together. The authors linked this secret cooperation to a classical idea called Simple Penal Codes and showed how to detect it by measuring differences in the programs’ choices. They created a new method called CURB that changes the rewards these programs get to stop them from secretly cooperating. Tests show CURB works well to prevent this kind of cheating in common economic games and can be used with different learning approaches.
What this means in practice
- •For platform designers: Stop covert collusion by AI agents managing pricing or competition dynamics in repeated economic settings using reward adjustments.
- •For machine learning engineers: Integrate a new reward shaping framework to reduce undesired collusive behavior in multi-agent reinforcement learning environments.
Authors
Karthik Sivachandran, Rohan Paleja
Abstract
Reinforcement learning agents trained to maximize their own reward in repeated interactions can converge to supra-competitive outcomes resembling explicit collusion, without communication or shared design. Existing mitigation approaches are largely tied to specific economic settings, like two-sided platforms and auctions, leaving open how to design interventions for general repeated games. We address this gap by formalizing the connection between empirical observations from prior work on Q-learning collusion and classical theory of Simple Penal Codes (SPCs). We show any non-trivial SPC induces a quantifiable conditional dependence in agents' policies, detectable via the total variation distance between an agent's action distributions across cooperation and defection histories. Building on this connection, we propose CURB (Collusion Unwinding via Reward shaping and Belief injection), a reward-shaping framework that penalizes this Total Variation (TV) distance signal during Q-learning and is guaranteed to convert any SPC fixed point of the dynamics into a trivial one, thus precluding collusive equilibria sustained by punishment threats. Empirically, CURB substantially reduces collusion by Q-learning agents in both Bertrand and Cournot Competition Repeated Games. We further demonstrate that CURB extends to deep Q-network agents in Bertrand competition, suggesting the mechanism generalizes beyond tabular Q-learning.