Small models learn continuously with synthetic self-distillation
Continually Post-training Small Models via Synthetic Self-distillation
Machine LearningArtificial Intelligence
Summary
When teaching computers new tasks, they often forget what they learned before, especially smaller models used on devices. The authors found this problem happens because the model's new learning data is too different from what it already knows. They created a method called Synthetic Self-Distillation that helps small models learn new things without forgetting old knowledge by using a slightly bigger model to guide the learning. This method improves learning and retention on various tests, making it easier for small models to keep up with new data over time.
What this means in practice
- •For mobile app developers: Improve on-device AI by updating small models continuously without losing previous skills using synthetic self-distillation.
- •For embedded systems engineers: Enable small AI models in low-latency environments to learn new tasks incrementally while maintaining performance on older ones.
Authors
Woosung Koh, Hyunsoo Lee, Kyungjae Lee, Haeju Park, Dahyun Lee, Dasol Hwang, Moontae Lee
Abstract
Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge. While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-following abilities. Consequently, SDFT fails on smaller models critical for low-latency, on-device use. To enable continual learning at smaller scales, we first identify the root cause of failure. We show that poor acquisition and forgetting correlate linearly with the post-trained model's distribution shift, revealing the necessity of in-distribution learning. We thus introduce Synthetic Self-Distillation (SynthSelf), imitating SDFT's in-distribution rewrite by outsourcing the task to a minimally larger, same-family aligner. Coupled with off-policy supervision and rejection sampling, this provides the high-quality supervision small models require without breaking their original distribution. Evaluations spanning 4 acquisition datasets (including a mixture) and 14 retention benchmarks show SynthSelf consistently outperforms 6 methods (including off- and on-policy distillation) across 3 small model scales. Extensive further analyses confirm SynthSelf paves the way toward effective small-model continual learning.