Papers for

automated tutoring developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Hierarchical sampling improves math reasoning in language models

HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments

Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a central approach for improving mathematical reasoning in language models, but long-form completions introduce a difficult credit-assignment problem: different parts of a solution trace may contribute unevenly to final correctness. Existing policyoptimization objectives for RLVR commonly apply importance-sampling correction at either the token level (GRPO, DAPO) or the sequence level (GSPO), imposing different granularities for assigning credit across a response. We introduce Hierarchical Importance-Sampling Policy Optimization (HISPO), a segment-level policy-optimization method that constructs rollout-time entropy-derived contiguous segments, assigns soft entropy-based saliency weights, and applies clipped importance-sampling correction at the segment granularity. This provides an intermediate correction unit between token-level GRPO/DAPO and sequence-level GSPO. We evaluate HISPO by fine-tuning Qwen3-1.7B-Base on mathematical reasoning tasks. Across six benchmarks, HISPO improves Pass@8 over the strongest baseline on all benchmarks and matches or exceeds the strongest baseline in Acc@8 on five of them. On AIME25, HISPO improves over GRPO by +3.75 Acc@8 and +3.78 Pass@8, and over GSPO by +2.50 Acc@8 and +1.27 Pass@8. These results suggest that segment-level correction is a promising granularity for RLVR in long-form mathematical reasoning.

Mon 14 SeptArtificial IntelligenceMachine Learning
The gist
Reinforcement learning helps language models improve at solving math problems by learning from rewards, but it's hard to figure out which parts of a long answer actually led to the final correct solution. The authors introduced HISPO, a method that breaks down answers into meaningful segments based on how uncertain the model was when generating each part, then assigns credit to those segments instead of individual words or whole answers. They tested HISPO on several math datasets and found it generally led to better performance than previous methods. This suggests that dividing answers into segments helps models learn more effectively from long math problem solutions.
Open → 2609.15471v1

CircuitLens finds problem difficulty rankings can mislead reinforcement learning

CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards

Abstract: Reinforcement learning with verifiable rewards (RLVR) is sensitive to which problems a model trains on, yet existing selection criteria--difficulty filtering, hand-curation, reward-trajectory scoring--assess data value as an intrinsic property of problems, independent of the model that will learn from them. We introduce Circuit Reasoning Score (CRS), a selection signal derived from 46 reasoning-sensitive attention heads identified via contrastive ablation, computed in a single forward pass on the frozen base model without reward labels or rollouts. CRS runs against the intuitive hypothesis that stronger reasoning-circuit engagement produces better training data: on Qwen2.5-Math-7B, the lowest-engagement decile improves over random selection on three medium-difficulty benchmarks (GSM8K +2.0 pp, OlympiadBench +1.6 pp, Minerva +2.9 pp), while the highest-engagement decile gains less and is indistinguishable from the middle decile. The advantage has boundary conditions: on a domain-curated pool no selection method separates from the others; at 1.5B scale the useful direction differs; and the lowest-reward training condition produces the strongest downstream generalization. Within the Qwen2.5-Math settings tested, RLVR data selection appears regime-dependent rather than reducible to a static ranking of problem quality.

Mon 7 SeptComputation and LanguageMachine Learning
The gist
Choosing which problems to train on is important for reinforcement learning with verifiable rewards (RLVR), but usual methods rank problems on their own merits, ignoring the specific model learning from them. The authors introduce a new way called Circuit Reasoning Score (CRS) that measures how much certain reasoning circuits in the model engage during problem solving. Surprisingly, the easiest problems by this measure sometimes lead to better learning outcomes than the hardest ones. Their experiments show that selecting training data based on reasoning circuit signals depends on the training setup and is not a fixed way to rank problem quality.
Open → 2609.07183v1