Large language models improve math reasoning with self-play data
Self-Play Search Distillation for Large Language Model Reasoning
Artificial Intelligence
Summary
Improving how language models solve math and reasoning problems needs lots of good practice examples. The authors created a way for AI systems to play games against themselves to generate high-quality reasoning steps without needing human help. They then use these examples to train language models, which helps them get better at solving new math problems. This method raises performance on math tests significantly and reduces the need for expensive human-labeled data.
What this means in practice
- •For ai model trainers: Create higher quality training datasets for reasoning tasks without costly human labeling by using self-play search data.
- •For automated math tutoring platforms: Improve automatic reasoning and solution generation capability in math problem solvers by leveraging LLMs trained on self-play generated data.
Authors
Lorenzo Molfetta, Wai-Chung Kwan, Giacomo Frisoni, Luca Ragazzi, Gianluca Moro, Pavlos Vougiouklis, Jeff Z. Pan, Pasquale Minervini
Abstract
Improving reasoning abilities in Large Language Models (LLMs) requires high-quality data that exposes difficult decisions, competing alternatives, and their consequences. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling. We introduce Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games. SPSD uses executable environments to turn search into structured reasoning problems. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates. By converting the self-play search records into superhuman chains-of-thought, we train LLMs with environment-grounded supervision. Although trained only on self-play search records, SPSD transfers to unseen mathematics. On Qwen3-4B-Base, it raises the mean over six mathematics benchmarks from 24.1 to 36.6 while increasing the held-out-game win rate from 15% to 45%. SPSD offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.