Cloopd improves on-policy distillation efficiency with adaptive learning passes
CLOOPD: Closing the Learner Loop in On-Policy Distillation
Machine Learning
Summary
Teaching AI models usually involves the student generating behavior and the teacher evaluating it, which can be inefficient. The authors introduce CLOOPD, a new way to separate scoring from learning updates, allowing multiple learning passes per scored set of data. This helps the student learn better from the teacher's feedback with fewer resources. Their tests show CLOOPD matches or outperforms traditional methods while using less computation and fewer teacher evaluations.
What this means in practice
- •For machine learning engineers: Improve training efficiency when transferring policies from teacher models to student models in reinforcement learning pipelines.
- •For cloud infrastructure teams: Reduce GPU usage and costly computing resources by adopting adaptive update scheduling in on-policy distillation workflows.
Authors
Keye Zheng, Hanyu Li, Zhan Cheng, Yuan Gao
Abstract
On-policy distillation (OPD) pays twice for each fresh batch: the student generates trajectories and a stronger teacher scores them. Existing methods improve which trajectories are scored and how the teacher signal is constructed, but usually consume it with one actor update. We introduce CLOOPD, a closed-loop framework separating teacher-signal acquisition from student-side realization. CLOOPD selects an adaptive $α$ waypoint inside a KL envelope, freezes the scored batch and its advantages, re-forwards the student after each actor pass, measures realization, and allocates actor work under a separate token budget. The framework includes deterministic two- and three-pass policies, token-priced CLOOPD-TPMR, and a budget-matched control. Across six 300-step runs on an 8-H20 node, every CLOOPD policy improves the one-pass TOP-D anchor at comparable teacher-token scale: macro accuracy rises from 15.41 to 17.78 with CLOOPD-Fixed2 and 19.36 with CLOOPD-Fixed3. At step 100, CLOOPD-Fixed3 reaches 15.35, nearly matching TOP-D at step 300 while using 67.2% fewer teacher-scored tokens and 28.0% fewer GPU-hours. Earlier 8-A100 ablations show adaptive $α$ eliminates observed trust-envelope violations; a third pass adds headroom. These results position CLOOPD as a framework for budgeting how fully students learn from teacher-scored tokens.