Asynchronous training speeds up language model self-learning by up to two times
Asynchronous On-Policy Self-Distillation
Machine Learning
Summary
Training language models to learn from themselves usually takes a long time because the process waits for every step to finish before moving on. The authors propose a method called AsyncOPSD that allows the model to train and generate examples at the same time without waiting. This speeds up training by about twice as much and improves the model's accuracy on math reasoning tasks. They also introduced ways to handle outdated training examples to keep the learning effective. This makes training large models more efficient without needing multiple separate models.
What this means in practice
- •For ai training engineers: Increase training speed and efficiency when refining language models using self-generated data in asynchronous workflows.
- •For machine learning infrastructure teams: Reduce system bottlenecks caused by synchronous rollout generation in large model training pipelines by adopting asynchronous approaches.
Authors
Xidong Wu, Qi Qi, Yuqiong Ji, Shangqian Gao
Abstract
Unlike on-policy distillation (OPD), which depends on a separate teacher model, on-policy self-distillation (OPSD) leverages a single language model acting simultaneously as the student and the teacher, with the teacher conditioned on privileged information. Despite this architectural simplicity, synchronous rollout generation dominates overall training time, creating a severe systems bottleneck when scaling OPSD. To address this, we introduce Asynchronous On-Policy Self-Distillation (AsyncOPSD), a framework that decouples rollout generation from learner updates to improve training throughput. Since the teacher and student share parameters, AsyncOPSD eliminates the multi-model coordination overhead inherent to OPD architectures in asynchronous training. As the student policy evolves, buffered trajectories generated by older policies become stale, potentially degrading performance. To mitigate this, we introduce a distillation objective based on staleness-aware policy control and token-level entropy weighting. Across three model sizes and mathematical reasoning benchmarks, AsyncOPSD improves the training throughput by $1.96$--$2.25\times$ over synchronous OPSD while increasing average benchmark accuracy.