Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
2026-07-21 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors address a problem in training language models to reason better when usual methods give no feedback if the model can't find any correct answers. They suggest giving the model a helpful hint during training, called off-context guidance, so it can learn from partly guided attempts even if the final goal is without hints. Their method, OC-GRPO, carefully adjusts for the difference between hinted and original problems to keep training stable. Tests show this approach improves performance on math reasoning tasks by a small but meaningful margin without extra cost.
reinforcement learninglarge language modelsverifiable rewardsguided rolloutsimportance correctionoff-context trainingmathematical reasoning benchmarksGRPO algorithm
Authors
Priyank Agrawal, Ankur Samanta, Shervin Ghasemlou, Jalaj Bhandari, Kavosh Asadi, Daniel Jiang, Aditya Modi
Abstract
Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models. Yet, typical RLVR approaches fail on difficult problems: when a model cannot generate any correct solutions, it receives \textit{zero} learning signal. Providing privileged guidance during training, such as solution prefixes, can help overcome this learning cliff by steering the model towards {correct solutions with non-zero reward}. {We call these rollouts \textit{off-context}: they are generated from a training prompt that contains privileged guidance, while the target objective is defined by the original prompt without that guidance.} {We introduce} Off-Context GRPO (OC-GRPO), a minimally modified variant of GRPO that uses guided rollouts but applies an importance-corrected objective to steer the update back toward the original unguided objective, avoiding the mismatch that destabilizes uncorrected guided training. Empirically, our algorithm achieves a 3.9\% absolute improvement (13.8\% relative gain) over vanilla GRPO on average across standard mathematical reasoning benchmarks with negligible additional cost.