One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL
2026-08-12 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors found that using just one large language model (LLM) to simulate human users in multi-agent reinforcement learning leads to poor performance when facing new or real users. They identified that the simulator LLM tends to focus on a narrow set of behaviors, causing the trained policy to overfit and not generalize well. To fix this, they introduced two methods: Verbalized Sampling, which makes the simulator’s responses more diverse during inference, and Co-Training, which trains the policy against multiple simulators to avoid overfitting. Their experiments showed these methods improved success rates on several benchmarks and also worked better with real users.
multi-agent reinforcement learninglarge language modelsimulator collapsemode collapsepolicy overfittingVerbalized SamplingCo-Trainingmulti-turn benchmarkspolicy diversitygeneralization
Authors
Simon Yu, Nicholas Tomlin, Marwa Abdulhai, Ximing Lu, Derek Chong, Abe Hou, Dilara Soylu, Sergey Levine, Christopher D. Manning, Weiyan Shi
Abstract
Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $τ^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.