Consolidating specialist AI models with no retraining needed
Training-Free On-Policy Distillation
Machine LearningArtificial IntelligenceComputation and Language
Summary
Combining AI models that are experts in different areas usually requires lots of retraining. This paper shows that by simply running these models on some example tasks and doing a special calculation, you can merge their strengths into one model without training. The combined model can perform better on average than the individual experts alone. The key trick is using a mathematical solution that matches each expert's responses layer by layer, based on data from their normal use.
What this means in practice
- •For machine learning engineers: Create a single model combining multiple specialized AI skills without costly retraining by using forward passes and least-squares fusion.
- •For enterprise ai platform teams: Deploy fused generalist AI models that maintain specialists’ performance across diverse tasks with just a few GPU-hours of forward computation.$Commercial implications: Offers a practical way to build and sell versatile AI services combining multiple expert models efficiently.
Authors
Bangji Yang, Jiajun Fan, Zhaoyu Zhu, Hongbo Ma, Ge Liu
Abstract
On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. One forward pass over a few hundred rollouts provides them. We call this training-free on-policy distillation (\method). What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. On Olmo~3 7B, a few GPU-hours of forward computation per specialist yield a generalist that keeps most of each specialist's gain on each of seven mathematics and code benchmarks.