Self-privileged critic improves intermediate value estimation in large language models
Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR
Machine LearningArtificial Intelligence
Summary
Training language models to reason step-by-step is hard because it’s tricky to decide which parts of the process deserve credit when only the final answer matters. The authors propose a new approach that helps the model’s critic compare current steps with past successful and unsuccessful attempts on the same question. This comparison makes the critic better at judging progress, which improves training and helps the model solve tough math problems more reliably.
What this means in practice
- •For ai model trainers: Improve training of language models on multi-step reasoning tasks by enhancing intermediate step evaluation.
- •For natural language processing engineers: Develop more reliable language model agents that better assess their reasoning progress during deployment.
Authors
Kun Liang, Chenming Tang, Clive Bai, Weijie Liu, Zeyuan Liu, Qingyang Zhang, Saiyong Yang, Yunfang Wu
Abstract
Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse terminal rewards, and actor-critic methods such as PPO address this by learning value functions to construct token-level advantages. Their effectiveness, however, hinges on reliable value estimation, a difficult task requiring the critic to both assess progress toward a correct solution and anticipate an evolving policy's future behavior; errors in either can compromise credit assignment and destabilize online training. In this paper, we revisit the standard state-only formulation of value estimation and propose $π$PPO, a self-privileged actor-critic framework. By reusing verified same-prompt rollouts as contrastive evidence, $π$PPO helps the critic assess intermediate reasoning against successful and failed attempts, while preserving standard policy optimization and the deployment interface. Experiments show that $π$PPO consistently improves value-estimation quality by a substantial margin and outperforms representative actor-critic and critic-free RLVR baselines on challenging mathematical reasoning benchmarks, while remaining effective even when paired with substantially smaller asymmetric critics.