AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Training
2026-06-01 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors present AlphaToken, a method to choose which response tokens (words or parts of words) to focus on when training large language models (LLMs) after their initial training. They break down token importance into two parts: adaptation, which helps the model learn new tasks, and stability, which keeps the model’s original abilities. AlphaToken uses signals from how tokens affect predictions along the generation path and approximates stability without needing original training data. By masking less important tokens during further training, their method helps improve performance and reduces forgetting old knowledge.
Token selectionLarge language modelsPost-trainingAdaptationStabilityGradientAutoregressive generationFisher informationCatastrophic forgettingGhost Dot-Product
Authors
Liu Qing, Ou Wu, Yi Du
Abstract
Token selection is pivotal for effective LLM post-training. However, existing methods mostly rely on local heuristics and rarely formulate token selection as a principled valuation of individual response tokens. We introduce $\textbf{AlphaToken}$, a response token valuation framework that decouples valuation into $\textbf{adaptation}$ (promoting target-task learning) and $\textbf{stability}$ (preserving pre-trained capabilities), and makes each objective $\textbf{path-aware}$ by combining the direct-path signal from local token gradients with the downstream causal-path signal in autoregressive generation. Since retention data are typically unavailable, AlphaToken approximates stability via a $\textbf{Fisher-drift proxy}$ anchored at the pre-trained reference model. For efficient computation, we extend Ghost Dot-Product to token-level valuation. AlphaToken masks low-value response tokens during fine-tuning and preference optimization, concentrating training signals on more valuable positions. Experiments show that AlphaToken improves post-training performance and mitigates catastrophic forgetting.