New training method improves click prediction accuracy with one network
FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction
Information RetrievalMachine Learning
Summary
Predicting whether people will click on ads or recommendations is hard because there are many ways features can interact. The authors studied a technique where a network learns from multiple versions of itself, each slightly different due to dropout, and found that this training approach implicitly encourages agreement between these versions. They designed a new model called FLMNet that uses this insight to train without needing to sample those versions randomly. Tests on five different tasks showed FLMNet often beats stronger models while only using one network when making predictions.
What this means in practice
- •For online advertising teams: Use FLMNet training to build single neural networks that predict user clicks more accurately without needing multiple models at runtime.$Commercial implications: Enables ad platforms to sell more effective single-model CTR prediction systems that are efficient and high-performing.
- •For recommendation engineers: Incorporate FLMNet to improve prediction accuracy in recommendation systems by distilling agreement from multiple training seeds into one model.
Authors
Yifan Wang, Yifei Du
Abstract
Click-through rate (CTR) prediction underpins recommender systems and online advertising, where high-dimensional categorical features interact in combinatorially many ways. Several recent CTR models, such as FinalNet and QNN, report higher accuracy when a network also learns from its own averaged predictions, for example by running the same network under different dropout patterns and training each run toward the average. Why this self-distillation helps, and whether its random sampling is needed, remains unclear. This work shows that as the number of dropout patterns grows without bound, the objective reduces exactly to the label loss of the logit-averaged prediction plus a penalty on how much the dropout copies of the network disagree. When dropout does not affect how a network gates its features, this penalty can be approximated in closed form, and FLMNet is designed on this principle and trains without sampling. Networks trained from different seeds disagree in the same form, and distilling several of them into one network recovers part of their ensemble gain. On five benchmarks against seventeen baselines, the resulting models take the top two places in accuracy with a single network at inference, and the gains of the penalty and of distillation, where measurable, largely add up.