Specialist model tuning controls student model expertise and general skills
Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation
Machine LearningComputation and Language
Summary
When teaching AI models specialized knowledge, the pathways they take to answer questions matter. This paper finds that even without explicit reasoning steps, the way specialists are trained strongly influences the knowledge students get from them. By analyzing how different training styles shift the specialists' focus between precise domain expertise and broader general knowledge, the authors show that these shifts are passed on to the student models. This means training choices can steer what kind of skill the student AI learns, even without clear reasoning guides.
What this means in practice
- •For natural language processing teams: Adjust training methods to balance domain-specific knowledge and general language understanding in AI assistants.
- •For scientific research model developers: Design AI models for chemistry or physics that reflect desired expertise profiles by tuning specialist training without detailed reasoning labels.
Authors
Yilei Tu, Zihao Li, Shaoxiong Ji, Jörg Tiedemann, Fei Yuan
Abstract
Specialist distillation effectively transfers domain expertise to student models via teacher-generated reasoning trajectories. However, when these specialists are trained solely on question--answer pairs without explicit reasoning supervision, what governs the trajectories they generate? In this work, we show that specialist optimization implicitly selects from this latent trajectory space. To isolate and observe this latent distribution, we leverage student distillation not as a downstream goal, but as an agnostic probe---since students inherit no parameterization or optimization constraints from the specialist, inheriting only the sampled trajectories themselves. Through this probe, our empirical analysis unveils a tight governing relationship: across 27 specialist--student pairings, their specialization--generalization profiles correlate exceptionally strongly. Crucially, explicitly controlling the specialist's distributional drift systematically shifts both the teacher and its distilled student along a controllable trade-off between domain precision and general-capability retention. Across chemistry, physics, and multilingual settings, distilled students systematically reflect these specialist-induced profiles, even across divergent model families. Our findings establish a new view of specialist training: when gold reasoning is absent, tuning choices directly control the latent supervision passed to downstream models.