Weak teacher model helps stronger model improve reasoning without retraining
Allspark: Weak to Strong Transfer via Alternating Chain of Thought
Machine LearningArtificial Intelligence
Summary
Large AI models can learn to reason better, but training them is expensive and slow. The authors introduce Allspark, a technique where a smaller, weaker model first thinks through parts of a problem and then helps a larger, stronger model finish the reasoning. This way, the big model benefits from the small model’s problem-solving skills without extra training on its own. They show this works across different AI model families and tasks involving math and reasoning.
What this means in practice
- •For machine learning engineers: Improve large model reasoning accuracy using a trained small model without retraining large models.
- •For ai system integrators: Combine outputs from different AI models by alternating reasoning steps to enhance final answers across diverse model types.
Authors
Kaizhao Liang, Junxiong Wang, Chen Liang, Zhendong Wang, Qiang Liu
Abstract
Recent progress in frontier models has renewed interest in large-scale reinforcement learning (RL), but the cost of generating large-model rollouts makes even testing RL recipes expensive. We ask whether reasoning improvements learned by a small, weak model can benefit a larger, stronger model without using the strong model's rollouts during training. We introduce Allspark, a training and inference framework for weak-to-strong transfer through alternating chains of thought. A weak teacher is trained alongside a frozen copy of the same model; the two alternate reasoning segments, and the frozen model produces the final answer. At inference time, a stronger student replaces the frozen training partner, while both models remain fixed. Because they communicate through text, the teacher can steer students from different model families and with different tokenizers. We study Allspark at two scales: controlled Qwen experiments across math and reasoning, and larger-scale Inkling experiments on ARC-AGI-2. The Inkling experiments show accuracy gains in within-family and cross-family settings, including transfer to Kimi and Nemotron, with benefits that vary across inference settings. These findings motivate reusing a trained weak teacher across strong students and examining the resulting accuracy--token tradeoff.