Rationale guided policy optimization improves reasoning in language models

Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding

Artificial Intelligence

Summary

Large language models can struggle to learn how to solve complicated problems because they rarely get clear feedback when they’re wrong. The researchers introduced a new way called Rationale-Guided Policy Optimization (RGPO), which helps models by giving them helpful explanations or hints temporarily while they learn. This guidance helps the model improve its answers, and once it does, the model learns directly from its own better answers without depending on the hints anymore. Testing showed RGPO helps models learn better reasoning with text and images compared to other methods.

What this means in practice

  • For language model developers: Improve reasoning abilities of language models by using adaptive rationale guidance during training to overcome sparse rewards.
  • For multimodal ai engineers: Enhance vision-language models’ problem-solving by integrating temporary rationale use for more effective reinforcement learning.

Authors

Hoang Phan, Minh Pham, Chau Pham, Chinmay Hegde, Trung Le, Qi Lei

Abstract

On-policy reinforcement learning has become a central paradigm for improving the reasoning abilities of large language models. However, its effectiveness is often limited by reward sparsity: when a model fails to discover correct trajectories for difficult problems, the optimization process receives little useful signal and may stagnate. Existing approaches mitigate this issue by incorporating off-policy demonstrations, expert traces, or model-generated solutions, but they typically require the auxiliary data to match the format of the reinforcement-learning task, often relying on rejection sampling from stronger models to obtain suitable training trajectories. We introduce Rationale-Guided Policy Optimization (RGPO), a framework that adaptively leverages ground-truth rationale information according to the model's current capability while preserving its freedom to explore. Rather than treating reference solutions as fixed imitation targets, RGPO uses them as temporary scaffolds: rationales help the model generate improved responses, after which only higher-reward, model-generated solutions are transferred back to the original unguided setting. This design allows training to exploit available ground-truth information without requiring off-policy data to follow the same format as the RL task. Across both language-only and vision-language reasoning settings, RGPO consistently improves performance over RLVR baselines, and ablation studies show that adaptive rationale guidance is a key contributor to these gains. These results suggest that RGPO offers a practical and general approach for reducing reward sparsity, stabilizing reinforcement learning, and improving reasoning performance in both text-only and multimodal models.