Diverse reasoning paths improve reinforcement learning performance in language models
Diverse Reasoning Paths Matter: Symbolic Equivalence Augmentation for Enhancing RL Exploration
Symbolic ComputationArtificial Intelligence
Summary
Getting AI models to think step-by-step helps them solve problems better. This paper finds that teaching them many different correct ways to solve the same problem helps even more. The authors created a method that changes reasoning steps while keeping the answers right, which makes the AI explore more creativity during learning. This leads to better reasoning after reinforcement training, meaning the AI can solve problems more accurately.
What this means in practice
- •For ai model developers: Improve fine-tuning data to increase reasoning diversity for better problem-solving performance after reinforcement learning.
- •For educational software engineers: Create AI tutors with stronger step-by-step reasoning by training on diverse, equivalent explanations.
Authors
Xiaoou Liu, Tiejin Chen, Hanyang Chen, Yuan Ling, Hua Wei
Abstract
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for improving the reasoning capabilities of LLMs. However, the role of supervised fine-tuning (SFT) data in enabling effective downstream RL remains underexplored. In this paper, we investigate a key hypothesis: the diversity of reasoning steps in chain-of-thought (CoT) during SFT is a critical enabler of RLVR performance. To test this hypothesis, we introduce EquivAug, a data augmentation framework that uses symbolic equivalence transformations to generate diverse yet provably correct reasoning paths. By converting CoT solutions into formal symbolic representations, applying equivalence transformations, and reconstructing natural-language explanations, EquivAug produces training data that differs in reasoning steps rather than surface-level lexical variation. Experiments on GSM8K and FOLIO across model scales demonstrate that EquivAug consistently achieves the highest reasoning path diversity after SFT and the strongest accuracy after GRPO. Our results establish that CoT diversity during SFT is positively correlated with downstream RLVR performance.