Per-token gating improves on-policy distillation in language tasks
A Unified Per-Token Gating Family for On-Policy Distillation: FKL/RKL Mixing with Multi-Channel and Bias Coefficients
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Teaching smaller AI models to learn from bigger ones can be tricky, especially at the level of individual words or tokens. The authors studied different ways to decide how much to trust the big model’s guidance for each token. They introduced a flexible method that blends several techniques and found it often leads to better performance on tasks like detecting emotions and hate speech. This approach can help make smaller models smarter by dynamically choosing when to listen to the teacher model.
What this means in practice
- •For natural language processing engineers: Improve the accuracy of smaller language models by implementing dynamic per-token gating during knowledge distillation from larger models.
- •For sentiment analysis developers: Enhance emotion and hate speech detection systems by using advanced on-policy distillation techniques to better align small model outputs with trusted large models.
Authors
Suwan Wu, Yumeng Lin, Pengcheng Yuan, Xiaolong Jiang
Abstract
Per-token gating of forward/reverse KL losses has become a standard technique for on-policy knowledge distillation (OPD), but existing methods such as EOPD (Jin et al., 2026) and ToDi (Jung et al., 2025) each fix a single gating signal and a single gating direction, and the two have never been compared directly. We introduce a four-coefficient parameterization lambda_t = sigma(a * h_t + b * u(x) + c + d * gap_t) in which direction-aligned proxies of EOPD and ToDi appear as one-dimensional (1D) restrictions, and which adds multi-channel composition and an explicit bias as further degrees of freedom. On TweetEval (Barbieri et al., 2020) emotion and hate, with a Qwen3-32B teacher and a Qwen3-4B student, configurations in the full family reach higher accuracy than the matched-magnitude single-channel (entropy-only / gap-only) 1D restrictions in 33 of 36 comparable cells, and a 26-cell mean-match isolation experiment places dynamic gating ahead of effective-KL-matched static baselines in 19 of 26 cells. Because cells share training data, models, and parameter substructure, we report both counts as exploratory aggregate directional evidence rather than as independent hypothesis tests. Targeted three-seed paired replications of the nine headline comparisons singled out by that sweep -- including a third task, offensive -- are directionally consistent, but individually smaller than the single-seed estimates and not significant at n=3. We therefore present the parameterization primarily as a shared coordinate system for comparing per-token gating designs in short-output classification OPD.