RREDCoT: Segment-Level Reward Redistribution for Reasoning Models
2026-06-04 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors look at how language models learn to solve problems step-by-step, called Chain-of-Thought (CoT). They point out a challenge where the model only gets feedback at the very end, making it hard to know which parts helped most. To fix this, the authors propose RREDCoT, a method that spreads out the rewards across important steps without needing extra slow computations. They compare their approach to other methods and study how best to split and evaluate the reasoning steps for training.
Reinforcement LearningChain-of-ThoughtReward RedistributionCredit AssignmentGroup Relative Policy OptimizationMonte Carlo MethodsLanguage ModelsState Value Estimation
Authors
Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger, Sepp Hochreiter
Abstract
Recent advancements in reasoning language models have been driven by Reinforcement Learning (RL) fine-tuning. Most often, these rely on the Group Relative Policy Optimization (GRPO) algorithm or modifications thereof to steer the models to produce Chain-of-Thought (CoT) traces. The final answer can only be verified, and the reward assigned, after the CoT trace is complete, making it a delayed reward problem. GRPO and its modifications correspond to Monte Carlo methods in standard RL, which are known to suffer from high variance. A possible solution to this problem is the redistribution of rewards through credit assignment, where segments of the CoT trace that are important for arriving at the desirable solution are emphasized by assigning a higher reward. While Monte Carlo sampling can be used to provide an unbiased estimate of intermediate state values, its computational overhead makes it unsuitable for train-time credit assignment in long contexts at high granularity. We introduce RREDCoT (Reward REDistribution for Chain of Thoughts), which utilizes the model itself to approximate the optimal reward redistribution without additional generation. We investigate the advantages of our method compared to MC sampling and several attribution methods. We further analyze several aspects relevant to the construction of the redistribution such as segmentation of CoT traces and state value estimation.