Multi-expert teaching improves scientific reasoning in language models
Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning
Artificial Intelligence
Summary
The paper looks at how to combine multiple expert AI teachers to train a student model better. Instead of giving all advice equally for an entire answer, the authors show it's better to pick which expert teaches each token based on how useful they are at that step. They propose a method that uses verification to decide which expert gives feedback on which part of an answer. This approach leads to better overall performance on scientific reasoning tests, especially for questions needing detailed knowledge. The key improvement comes from directing the advice precisely rather than just adding more teachers.
What this means in practice
- •For language model developers: Enable training of student models that learn selectively from multiple expert models at a fine-grained level to improve reasoning accuracy on scientific tasks.
- •For automated scientific assistants: Improve the accuracy of AI systems that assist with scientific question answering by integrating specialized expert capabilities efficiently.
Authors
Xun Xu, Zaixi Zhang
Abstract
Multi-teacher on-policy distillation (OPD) is becoming the standard way to integrate specialist capabilities into one model: train experts with RL, then distill them into the student on its own rollouts. Existing recipes assign supervision at the sequence level - each prompt goes to one domain teacher and every token receives the same weight - which implicitly assumes that a teacher is uniformly useful across a response. We find instead that useful teacher signal is sparse and heterogeneous along a reasoning trajectory, which raises a finer question: who should teach which token? Verifier-Gated Multi-Expert On-Policy Distillation (VG-OPD) answers it by verification: the counterfactual gain of an expert on a specific answer criterion licenses that expert to teach, its disagreement with the student localizes the supervision, and criterion importance sets its weight; the gated KL enters GRPO as an additive token-level advantage. Instantiated for scientific reasoning with RL-trained capability experts, VG-OPD attains the best overall performance on seven benchmarks for 4B and 8B students, ranking first on five at both scales, with the largest gains on knowledge-intensive scientific reasoning tasks. Further analysis shows that the gains come from localizing verified supervision rather than from adding teachers or distillation loss: misplacing the same supervision budget is the single most damaging change, and indiscriminate distillation drags RL below its own floor where gated distillation lifts it.