Summary
Choosing the right problems for a computer to learn from is important but tricky because different problems help different models in different ways. The authors looked at special signals inside a trained AI model that show when it is reasoning deeply. They found that picking training problems where the model’s reasoning is less active sometimes leads to better learning results than picking problems where reasoning is strongest. However, this pattern changes depending on the model size and problem set, meaning there is no one best way to pick problems. Their work shows that how well a set of training problems works depends on the learning situation, not just on the problems themselves.
reinforcement learningverifiable rewardstraining data selectionattention headscontrastive ablationreasoning circuitsmodel scaleQwen2.5-Mathbenchmarkinggeneralization
Authors
Zhuofan Chen, Ziqian Jiao, Yikai Cui, Zhixin Cai, Jun Bai, Wenge Rong
Abstract
Reinforcement learning with verifiable rewards (RLVR) is sensitive to which problems a model trains on, yet existing selection criteria--difficulty filtering, hand-curation, reward-trajectory scoring--assess data value as an intrinsic property of problems, independent of the model that will learn from them. We introduce Circuit Reasoning Score (CRS), a selection signal derived from 46 reasoning-sensitive attention heads identified via contrastive ablation, computed in a single forward pass on the frozen base model without reward labels or rollouts. CRS runs against the intuitive hypothesis that stronger reasoning-circuit engagement produces better training data: on Qwen2.5-Math-7B, the lowest-engagement decile improves over random selection on three medium-difficulty benchmarks (GSM8K +2.0 pp, OlympiadBench +1.6 pp, Minerva +2.9 pp), while the highest-engagement decile gains less and is indistinguishable from the middle decile. The advantage has boundary conditions: on a domain-curated pool no selection method separates from the others; at 1.5B scale the useful direction differs; and the lowest-reward training condition produces the strongest downstream generalization. Within the Qwen2.5-Math settings tested, RLVR data selection appears regime-dependent rather than reducible to a static ranking of problem quality.