Large language models improve themselves using synthetic questions and answers
On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Large language models usually need external examples or human help to get better after training. This work shows that models can improve by creating their own practice questions and answers based only on general topics, without any human-provided examples or corrections. The models generate both the questions and detailed reasoning steps themselves, and then learn from these synthetic data. This approach helps the models get better in different areas while keeping their original skills, and it works for both text and images.
What this means in practice
- •For language model developers: Improve language model performance after initial training without requiring labeled data or external supervision.
- •For multimodal ai engineers: Enhance models that handle both text and images by enabling self-improvement using only synthetic self-generated data.
Authors
Leon Sick, Lukas Hoyer, Yannick Strümpler, Christian Reisswig, Timo Ropinski, Federico Tombari
Abstract
On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.