Self-improving reasoning models sustain gains by filtering questions
Questioning the Questions: Sustaining Self-Evolution in Reasoning Models
Machine LearningArtificial IntelligenceComputation and Language
Summary
Self-learning AI models can get worse when they keep practicing with their own questions because some questions are invalid or repeated in different ways. The authors studied these problems and created a method called R-Quest that teaches the model to spot bad questions and encourages it to generate new ones. This approach helps the model improve steadily over many rounds of training in tasks like math, reasoning, and coding. It also performs better than earlier methods by a notable margin.
What this means in practice
- •For ai development teams: Improve reasoning and coding AI systems by integrating question validation and novelty checks to maintain steady learning progress.
- •For automated tutoring systems: Enhance the quality of generated practice questions by filtering out invalid or repetitive items to support better student learning experiences.
Authors
Jinyuan Li, Chengsong Huang, Langlin Huang, Donghong Cai, Shiping Gao, Yuyi Yang, Jiaxin Huang
Abstract
Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we investigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self-generated questions: invalid questions and repeated variants of the same mathematical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical reasoning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforming R-Zero by 17.32 points.