GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
2026-08-31 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors studied why certain parts of text, called high-entropy tokens, help improve language models when training them with reinforcement learning. They found that just looking at how surprising a token is (entropy) doesn’t always show how important it is for learning. Instead, they created a new method called Gradient Magnitude-based Token Selection (GMTS) that better estimates token importance by using how much the model’s parameters change during training. Their experiments showed that training with tokens chosen by GMTS works better than using entropy alone across different tasks and model sizes.
Reinforcement learningLarge language modelsEntropyToken importanceGradient magnitudeToken selectionReinforcement Learning with Verifiable Rewards (RLVR)Model trainingReasoning tasksGradient-based methods
Authors
Outongyi Lv, Yuanwei Zhang, Xiaoqun Zhang
Abstract
Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language models' (LLMs) reasoning abilities, demonstrating remarkable effectiveness across reasoning tasks. Recent studies suggest that high-entropy tokens play an exceptionally important role in model training, since training with only the highest 20% entropy tokens yields significant performance gains. However, why such high-entropy tokens are beneficial remains insufficiently understood. In this work, we find that although high-entropy tokens within one answer tend to correlate with large gradient magnitude, entropy alone fails to consistently reflect token importance across different answers, considering the variations in the answer-level reward signals. Based on this observation, we introduce the Gradient Magnitude-based Token Selection (GMTS) method to quantify token importance, which leverages the entropy-gradient connection to approximate gradient-magnitude rankings for token selection. We find that training on the top 20% tokens ranked by GMTS consistently outperforms entropy-based token selection across three reasoning domains and various model sizes, suggesting that GMTS provides a more fine-grained estimate of token contribution for RLVR training.