Cancellation-aware masking improves reinforcement learning in large language models

CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning

Machine LearningArtificial IntelligenceComputation and Language

Summary

Training large language models with reinforcement learning can be tricky because updates sometimes conflict with the way responses are generated. The authors found that the usual method for deciding which responses to learn from can miss important differences due to balancing effects in the math. They created a new masking method called CARM that catches these hidden differences better by looking at token changes without letting opposing effects cancel out. CARM helps the model learn more effectively and improves performance on math problems and code writing.

What this means in practice

  • For software development teams: Improve code generation quality by integrating CARM to better guide large language models during reinforcement learning updates.
  • For ai model trainers: Enhance mathematical reasoning skills of large language models using CARM for more reliable reinforcement learning with off-policy responses.

Authors

Yafei Zhang, Songshuo Lu, Sicong Liao, Zhi Chen, Yaohua Tang

Abstract

Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift. We propose \emph{Cancellation-Aware Response Masking} (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to $3.13$ percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by $2.88$ points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning.