Teacher and student models trained separately improve semi supervised learning
Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student
Machine Learning
Summary
Semi-supervised learning tries to teach computers with only a few labeled examples and many unlabeled ones. Usually, a teacher model helps a student model learn, but they share many settings, which can cause problems in learning. The authors found these problems come from the teacher and student being too tightly linked and the combined learning signals being unbalanced. They introduced a new helper model called the Pioneer Student that learns separately and occasionally shares its knowledge back, making the whole system learn better. This helper can be added to many existing methods to improve them.
What this means in practice
- •For machine learning engineers: Integrate an independent helper model to improve training stability and generalization in semi-supervised learning tasks that use teacher-student frameworks.
- •For computer vision teams: Enhance semi-supervised image recognition systems by reducing overfitting and boosting performance via decoupled optimizer branches.
Authors
Haorong Han, Jidong Yuan, Chixuan Wei, Yongqi Sun
Abstract
Semi-supervised learning (SSL) relies on two core mechanisms: self-training under the Teacher-Student (T-S) framework and joint optimization of labeled and unlabeled losses. Despite their effectiveness, we find both mechanisms introduce distinct optimization pathologies. First, parameter coupling enforces strict synchronization between teacher and student, where strong regularization on the student degrades the teacher's fitting ability, thereby limiting the permissible generalization intensity. Second, the imbalance in gradient update consistency between labeled and unlabeled losses drives the shared parameters to prematurely converge to labeled-dominated local minima, creating a bottleneck for global optimization. To address both issues, we propose the Pioneer Student (PiS), an auxiliary branch that operates in an independent parameter space and periodically transfers accumulated knowledge back to the T-S model. Extensive experiments show that PiS is a universal plug-and-play module that consistently improves mainstream SSL methods.