Kl-mopd improves multi-goal token-by-token recommendation learning
KL-MOPD: Discrepancy-Aware Multi-Objective On-Policy Distillation for Generative Recommendation
Information Retrieval
Summary
Generative recommendation systems suggest items by predicting one word at a time, making it tricky to balance different goals like clicks and purchases simultaneously. The authors looked at combining advice from two separate models focused on one goal each, but found it was hard for the student model to learn both well at every step. They created KL-MOPD, a way to weigh which teacher model’s guidance should be trusted more for each word, based on how different the student is from the teacher. This helps the recommendation system make better choices for both goals and performs better in tests on public and industrial data.
What this means in practice
- •For e-commerce platform engineers: Enhance recommendation engines to better balance user clicks and purchases by using token-level weighted distillation from multiple teacher models.
- •For advertising technology teams: Improve ad recommendation quality by integrating KL-MOPD to jointly optimize for multiple campaign objectives in token-based generative models.
Authors
Weiqin Yang, Yandong Wang, Yuan Gao, Hao Deng, Jinxin Hu, Jing Cao, Jiawei Chen
Abstract
Coordinating multiple recommendation objectives (e.g., click and conversion) is critical for real-world generative recommendation (GR). However, since a positive item in GR can be retrieved only when its tokens are successively selected at every decoding step, traditional item-wise methods lack fine-grained token-wise supervision to guide intermediate steps. Motivated by its success in foundation models, we consider introducing multi-objective on-policy distillation (MOPD) into GR to distill two single-objective teachers into a student at each token. Yet, we find that in vanilla MOPD, (i) teachers exhibit unequal distillation difficulty at each token (i.e., token-level imbalance), and (ii) the harder-to-distill teacher varies across tokens (i.e., cross-token inconsistency), leaving each objective poorly learned at certain steps and hindering positive item retrieval for both objectives, leading to suboptimal performance. To address this challenge, we propose KL-MOPD (KL-Weighted Multi-Objective On-Policy Distillation), a simple yet effective distillation method that allocates token-wise teacher weights proportional to the initial student-teacher token-wise KL divergences, thereby strengthening supervision for the harder-to-distill teacher at every decoding step. Despite its simplicity, we theoretically prove that KL-MOPD strictly outperforms vanilla MOPD in the additive joint recommendation objective under mild conditions. Moreover, KL-MOPD is plug-and-play and computationally efficient, requiring only a one-time pre-training weight calibration with negligible training overhead. Extensive experiments on three public benchmarks, a billion-scale industrial dataset, and online A/B tests on our e-commerce advertising platform demonstrate that KL-MOPD significantly improves both individual and joint recommendation performance. The anonymous code is provided for peer review.