DAPD: Dual-Anchored Policy Distillation
2026-08-03 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors study a problem in training language models where a student model learns from a teacher model that has extra information not available during actual use. This mismatch causes the student to perform worse since it expects information it won't get later. They propose a new method called Dual-Anchored Policy Distillation (DAPD) that creates two kinds of connections to make sure the student only learns what it can use at inference time. Their experiments show DAPD improves performance and reduces this reliance on privileged information, working well across different model sizes.
On-policy distillationLanguage modelsTeacher-student learningPrivileged informationInference-time contextPolicy distillationDual-Anchored Policy DistillationSelf-conditioningRolloutModel scaling
Authors
Jianyu Wu, Yizhou Wang, Encheng Su, Chen Tang, Shixiang Tang
Abstract
On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.