Language models improve few-shot learning with dynamic training shifts
PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation
Machine LearningArtificial Intelligence
Summary
Small language models often struggle to learn from only a few examples in specific topics or areas. The authors introduce a method called PIVOT that decides when the model should learn by copying a teacher's guidance and when it should try improving by itself using rewards, based on how confusing the teacher finds the data. This approach lets the model spend the right amount of time learning from the teacher or exploring on its own for better results. Tests show PIVOT leads to better accuracy and more stable training compared to fixed schedules.
What this means in practice
- •For customer support developers: Enhance intent classification models with adaptive training schedules to better handle limited labeled examples in banking and service domains.
- •For chatbot designers: Improve chatbot domain adaptation by applying dynamic knowledge distillation combined with reinforcement learning for more accurate responses on few-shot data.
Authors
Heng Li, Yong Zhang, Ning Cheng, Zhigen Li, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao
Abstract
Vertical-domain few-shot classification remains challenging for small language models, as limited supervision makes it difficult to acquire domain-specific decision knowledge. On-Policy Distillation (OPD) can improve teacher-guided adaptation by supervising student-generated rollouts, while GRPO-based reinforcement learning can further refine downstream predictions. However, existing KD-to-RL pipelines typically rely on globally fixed transition schedules, ignoring that different samples may require different amounts of teacher-guided acquisition before reward-driven refinement. We propose PIVOT (Perplexity-Informed Transition Optimization), a dynamic transition framework that routes samples between OPD and GRPO according to teacher-evaluated sequence perplexity. PIVOT moves low-perplexity samples to GRPO for reward-driven refinement while keeping high-perplexity samples under OPD for continued domain knowledge acquisition. Experiments on Banking77 and HWU64 show that PIVOT consistently outperforms continued OPD and globally synchronized OPD$\rightarrow$GRPO baselines under the same number of post-warm-up student optimization steps, achieving stronger downstream performance and more stable training dynamics.