CircuitLens finds problem difficulty rankings can mislead reinforcement learning
CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards
Summary
Choosing which problems to train on is important for reinforcement learning with verifiable rewards (RLVR), but usual methods rank problems on their own merits, ignoring the specific model learning from them. The authors introduce a new way called Circuit Reasoning Score (CRS) that measures how much certain reasoning circuits in the model engage during problem solving. Surprisingly, the easiest problems by this measure sometimes lead to better learning outcomes than the hardest ones. Their experiments show that selecting training data based on reasoning circuit signals depends on the training setup and is not a fixed way to rank problem quality.
What this means in practice
- •For machine learning engineers: Select training problems by analyzing model reasoning circuit engagement to potentially improve learning efficiency and generalization within specific training regimes.
- •For automated tutoring developers: Design adaptive problem sets that consider how students’ reasoning engagement patterns might affect learning outcomes, informed by model-based data selection insights.