CircuitLens finds problem difficulty rankings can mislead reinforcement learning

CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards

Computation and LanguageMachine Learning

Summary

Choosing which problems to train on is important for reinforcement learning with verifiable rewards (RLVR), but usual methods rank problems on their own merits, ignoring the specific model learning from them. The authors introduce a new way called Circuit Reasoning Score (CRS) that measures how much certain reasoning circuits in the model engage during problem solving. Surprisingly, the easiest problems by this measure sometimes lead to better learning outcomes than the hardest ones. Their experiments show that selecting training data based on reasoning circuit signals depends on the training setup and is not a fixed way to rank problem quality.

What this means in practice

  • For machine learning engineers: Select training problems by analyzing model reasoning circuit engagement to potentially improve learning efficiency and generalization within specific training regimes.
  • For automated tutoring developers: Design adaptive problem sets that consider how students’ reasoning engagement patterns might affect learning outcomes, informed by model-based data selection insights.

Authors

Zhuofan Chen, Ziqian Jiao, Yikai Cui, Zhixin Cai, Jun Bai, Wenge Rong

Abstract

Reinforcement learning with verifiable rewards (RLVR) is sensitive to which problems a model trains on, yet existing selection criteria--difficulty filtering, hand-curation, reward-trajectory scoring--assess data value as an intrinsic property of problems, independent of the model that will learn from them. We introduce Circuit Reasoning Score (CRS), a selection signal derived from 46 reasoning-sensitive attention heads identified via contrastive ablation, computed in a single forward pass on the frozen base model without reward labels or rollouts. CRS runs against the intuitive hypothesis that stronger reasoning-circuit engagement produces better training data: on Qwen2.5-Math-7B, the lowest-engagement decile improves over random selection on three medium-difficulty benchmarks (GSM8K +2.0 pp, OlympiadBench +1.6 pp, Minerva +2.9 pp), while the highest-engagement decile gains less and is indistinguishable from the middle decile. The advantage has boundary conditions: on a domain-curated pool no selection method separates from the others; at 1.5B scale the useful direction differs; and the lowest-reward training condition produces the strongest downstream generalization. Within the Qwen2.5-Math settings tested, RLVR data selection appears regime-dependent rather than reducible to a static ranking of problem quality.