Self-privileged critic improves intermediate value estimation in large language models

Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR

Machine LearningArtificial Intelligence

Summary

Training language models to reason step-by-step is hard because it’s tricky to decide which parts of the process deserve credit when only the final answer matters. The authors propose a new approach that helps the model’s critic compare current steps with past successful and unsuccessful attempts on the same question. This comparison makes the critic better at judging progress, which improves training and helps the model solve tough math problems more reliably.

What this means in practice

Authors

Kun Liang, Chenming Tang, Clive Bai, Weijie Liu, Zeyuan Liu, Qingyang Zhang, Saiyong Yang, Yunfang Wu

Abstract

Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse terminal rewards, and actor-critic methods such as PPO address this by learning value functions to construct token-level advantages. Their effectiveness, however, hinges on reliable value estimation, a difficult task requiring the critic to both assess progress toward a correct solution and anticipate an evolving policy's future behavior; errors in either can compromise credit assignment and destabilize online training. In this paper, we revisit the standard state-only formulation of value estimation and propose $π$PPO, a self-privileged actor-critic framework. By reusing verified same-prompt rollouts as contrastive evidence, $π$PPO helps the critic assess intermediate reasoning against successful and failed attempts, while preserving standard policy optimization and the deployment interface. Experiments show that $π$PPO consistently improves value-estimation quality by a substantial margin and outperforms representative actor-critic and critic-free RLVR baselines on challenging mathematical reasoning benchmarks, while remaining effective even when paired with substantially smaller asymmetric critics.