Projecting optimizer updates boosts multi-domain on-policy distillation accuracy
UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation
Computer Vision and Pattern Recognition
Summary
Training a single AI model to learn from different expert teachers can cause conflicting training signals that hurt learning. The authors found that common methods to fix these conflicts don’t work well with modern optimizers like AdamW. They propose a new method that adjusts the updates right before the model changes, ensuring no increase in errors from any domain. This new approach improved accuracy across tests involving medical, math, coding, and instruction tasks compared to previous methods.
What this means in practice
- •For machine learning engineers: Integrate multi-domain expert models into a single student model with fewer performance conflicts using the authors’ update projection method.
- •For healthcare ai developers: Combine medical knowledge with general domain data effectively in AI models for improved diagnostic and clinical decision support.
Authors
Taojie Zhu, Jing Jin, Yuan Xia, Chenyang Ding, Qunshan He, Wanke Xia, Tao Sun, Yan Chen, Jian Wang, Jinjie Gu, Tao Feng
Abstract
On-policy distillation from multiple teachers combines expertise from different domains in a single student, but conflicting gradients can hinder this integration. Gradient corrections directly constrain parameter updates under plain SGD. With optimizers such as AdamW, however, momentum, adaptive scaling, and weight decay can turn a corrected gradient into an update that increases a domain loss to first order. To address this gap, we propose Update Projection for Multi-Teacher On-Policy Distillation (UP-MOPD). UP-MOPD lets the original mixed gradient update the optimizer state and generate a candidate displacement, then projects only violating candidates before they are committed to the parameters. The projection gives the unique feasible update closest to the candidate in Euclidean distance. In experiments combining medical and general domains, UP-MOPD improves IFEval-loose accuracy late in training by 2.96 points over vanilla M-OPD. It achieves an average score of 60.03 across eight metrics, compared with 59.00 for gradient projection and 59.15 for update rejection. On a public benchmark covering mathematics, code, and instruction following, it achieves the best average across six tasks (32.67), leads on LiveCodeBench v5, and ties for the best IFEval result.These results support projecting optimizer updates to reduce interference between domains.