Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation
2026-08-10 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors find that traditional reinforcement learning methods struggle to learn from groups of examples when those groups are all correct or all wrong. To fix this, they introduce SKALD, a new method that trains a student model using guidance from a teacher model with extra skill information, but without needing that extra info during testing. This approach helps the student learn useful skills even when the usual reward signals are weak or unhelpful. Testing on math problems shows SKALD outperforms previous methods, especially with larger models, demonstrating that abstract skills can provide more useful feedback.
reinforcement learningself-distillationQwen3-Base modelskill cardteacher-student modelcross-entropyforward-KL divergencemathematics benchmarksrollout groupscontext distribution mismatch
Authors
Yubo Jiang, Fengying Xie, Zhiguo Jiang, Haopeng Zhang
Abstract
Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63.0-68.0% of groups in our experiments. We propose SKALD (Skill-Anchored Latent Distillation), an on-policy self-distillation framework that uses two context views of the same Qwen3-Base model: a question-only student and a teacher conditioned on an abstract, explicit-answer-filtered skill card. The student is trained on its own prefixes, transferring the skill-induced advantage into shared parameters without privileged input at test time. To stabilize context-induced distribution mismatch, SKALD employs an annealed exponentially tilted objective that downweights teacher-preferred tokens with very low student likelihood; as the tilt vanishes, it converges to teacher cross-entropy and recovers the forward-KL student gradient. An empirical gate activates distillation only when verified rollouts estimate a positive teacher advantage. Across five held-out mathematics benchmarks, SKALD improves overall avg@8 over GRPO by +2.46, +4.85, and +12.01 at 0.6B, 1.7B, and 4B, respectively. At 1.7B, zero-variance-only distillation recovers 84.7% of the full gain, while SKALD remains +4.06 above FLOP-matched GRPO and exceeds contextual skill exposure by +3.77. These results show that abstract skills provide dense supervision where group-relative rewards become uninformative.