Knowledge distillation improves click conversion predictions in recommendation systems
Serving-Aligned Entire-Space Conversion Modeling via Click-Through Rate Knowledge Distillation
Machine Learning
Summary
Predicting how often users will click and then buy something is important for recommendation systems. The authors found that training models using information from the actual click predictions made during service (called knowledge distillation) helps make better and more consistent conversion rate predictions. This method leads to more accurate rankings of items that users are likely to click and purchase. Tests on a large e-commerce dataset show this approach improves prediction quality compared to previous methods.
What this means in practice
- •For e-commerce engineers: Enhance conversion prediction accuracy in product ranking algorithms by aligning training with serving-time click rate estimates using knowledge distillation.
- •For advertising platform teams: Improve ad selection quality by reducing discrepancies between model training and online serving predictions of click and conversion rates using logged click data.
Tested on one dataset.
Authors
Pavel Dolin, Xun Zou, Jiatian Wu, Deguang Kong
Abstract
Recommendation systems may use predicted click-through conversion rate (pCTCVR), the product of predicted click-through rate (pCTR) and post-click conversion rate (pCVR), to guide ranking decisions. Many pCVR models build on the Entire Space Multi-Task Model (ESMM), which jointly trains pCVR and auxiliary pCTR towers using pCTR and pCTCVR losses, but deploys only the pCVR tower. At serving time, this tower may be paired with a separately trained pCTR model, creating a training-serving mismatch in pCTCVR when its predictions differ from those of the auxiliary pCTR tower. In this paper, we use knowledge distillation (KD) from logged serving pCTR scores to supervise ESMM's auxiliary pCTR tower and improve training-serving pCTCVR alignment. We denote variants by KD-$α$, where $α$ controls the hard-label loss weight. On the Ali-CCP dataset, KD-0.3, the best variant on serving pCTCVR, increases ROC-AUC not only for serving pCTCVR but also for pCVR by 2.1% and 1.6%, respectively, and PR-AUC by 9.9% and 8.8%, respectively, relative to the ESMM baseline. We also test negative downsampling with 5% nonclick retention. Downsampled KD-0.3 increases mean AUC for serving pCTCVR and pCVR by 1.2% and 0.9%, respectively, -and PR-AUC by 8.5% and 4.5%, respectively, relative to downsampled ESMM.