Le Critique: Privileged Value Functions for LLM Reinforcement Learning
2026-08-17 • Machine Learning
Machine Learning
AI summaryⓘ
The authors study ways to improve reinforcement learning for Large Language Models by reducing the randomness (variance) in learning signals. They point out that current methods either use multiple tries per prompt but only reward final outputs or use learned value functions for token-level feedback but are complex to implement. They introduce two ideas: Privileged Value Functions, which add helpful token-level signals without messing up learning, and TETHER, a method that smartly combines different baselines based on how good the value estimate is. Their methods perform better than standard value functions and are as good or better than popular group-based approaches on reasoning tasks.
Reinforcement LearningLarge Language ModelsVariance ReductionGroup-Relative Policy OptimizationValue FunctionsToken-Level AdvantagePolicy GradientBaselinesOff-PolicynessCredit Assignment
Authors
Siddarth Venkatraman, Matthieu Dinot, Laurence Aitchison
Abstract
Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy. Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit. Training is also blocked by straggler rollouts, reducing throughput and increasing off-policyness. Learned value functions theoretically address both problems, providing token-level advantages without requiring large groups. However, additional infrastructure engineering challenges combined with the practical success of critic-free methods have made it difficult to justify their inclusion in RL pipelines. We propose two complementary strategies to improve the performance of value function RL: 1) Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; 2) TETHER, a baseline that adaptively interpolates between group-relative and value baselines depending on the value function accuracy. Across several reasoning tasks, both strategies consistently improve over the standard value function baseline, and are competitive with or outperform mean-baseline GRPO.