Summary
Reinforcement learning helps language models improve at solving math problems by learning from rewards, but it's hard to figure out which parts of a long answer actually led to the final correct solution. The authors introduced HISPO, a method that breaks down answers into meaningful segments based on how uncertain the model was when generating each part, then assigns credit to those segments instead of individual words or whole answers. They tested HISPO on several math datasets and found it generally led to better performance than previous methods. This suggests that dividing answers into segments helps models learn more effectively from long math problem solutions.
What this means in practice
- •For machine learning engineers: Fine-tune language models more effectively on long reasoning tasks using segment-based importance sampling for better credit assignment.
- •For automated tutoring developers: Improve automated math tutors’ feedback quality by training models to better evaluate and generate stepwise solutions with segment-aware learning.
Abstract
Reinforcement learning with verifiable rewards (RLVR) has become a central approach for improving mathematical reasoning in language models, but long-form completions introduce a difficult credit-assignment problem: different parts of a solution trace may contribute unevenly to final correctness. Existing policyoptimization objectives for RLVR commonly apply importance-sampling correction at either the token level (GRPO, DAPO) or the sequence level (GSPO), imposing different granularities for assigning credit across a response. We introduce Hierarchical Importance-Sampling Policy Optimization (HISPO), a segment-level policy-optimization method that constructs rollout-time entropy-derived contiguous segments, assigns soft entropy-based saliency weights, and applies clipped importance-sampling correction at the segment granularity. This provides an intermediate correction unit between token-level GRPO/DAPO and sequence-level GSPO. We evaluate HISPO by fine-tuning Qwen3-1.7B-Base on mathematical reasoning tasks. Across six benchmarks, HISPO improves Pass@8 over the strongest baseline on all benchmarks and matches or exceeds the strongest baseline in Acc@8 on five of them. On AIME25, HISPO improves over GRPO by +3.75 Acc@8 and +3.78 Pass@8, and over GSPO by +2.50 Acc@8 and +1.27 Pass@8. These results suggest that segment-level correction is a promising granularity for RLVR in long-form mathematical reasoning.