Advantage based control variates improve reinforcement learning efficiency
ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards
Machine Learning
Summary
Reinforcement learning teaches computers to make decisions by rewarding good actions. The authors found that common methods sometimes add bias because they learn value estimates that aren’t perfect. They invented a new way to reduce this bias, called Advantage-Based Control Variates (ABC), which builds on existing ideas about advantages and values in learning. Their approach helps learning algorithms perform well using fewer trial-and-error steps, especially when rewards can be verified, such as in math reasoning tasks.
What this means in practice
- •For machine learning engineers: Improve efficiency of reinforcement learning agents adapting from offline data to online tasks with verified rewards.
- •For automated theorem proving teams: Develop reinforcement learning systems that reason about mathematical problems using fewer learning steps and verified feedback.
Authors
Hsiao-Ru Pan, Florent Draye, Bernhard Schölkopf
Abstract
Recent progress in reinforcement learning with verifiable rewards (RLVR) has highlighted the effectiveness of simple critic-free policy-gradient methods such as Group Relative Policy Optimization (GRPO). In contrast, actor-critic methods rely on learned value functions whose approximation error can introduce bias through commonly used advantage estimators such as temporal-difference error. Motivated by this observation, we revisit trajectory-level control variates through an advantage-value formulation, which we call Advantage-Based Control Variates (ABC). This formulation reveals that the covariance structure is closely related to the return decomposition used in Direct Advantage Estimation (DAE). Finally, we combine ABC with DAE into a single actor-critic algorithm and evaluate it in an offline-to-online RLVR setting, where the critic is first trained on previously collected trajectories and adapted during online learning. On mathematical reasoning tasks, ABC achieves performance competitive with GRPO using substantially fewer online optimization steps.