Transformers enhanced for large scale industrial recommendation systems
LazFormer: Scaling Transformers for Industrial Recommendation via Transferable Generative Pre-training
Information Retrieval
Summary
Recommender systems that suggest items to users can use Transformers, a type of AI model that works well with large data. The authors found that training these models from scratch is slow and costly, so they created a way to pre-train parts of the model to make training faster and better. They solved problems with transferring trained parts and overfitting by designing new components that adapt the model to specific recommendations and handle long user history efficiently. Their approach helps industrial recommendation systems learn effectively from large user data sequences while using computational resources wisely.
What this means in practice
- •For e-commerce platform engineers: Optimize recommendation models to handle long user histories efficiently with reduced training time and improved parameter transfer.
- •For online advertising teams: Deploy scalable recommendation systems that maintain adaptability to evolving user behavior without overfitting sparse model components.
Authors
Xiaodong Li, Alin Fan, Mingyang Li, Yan Xiao, Shichao Nie, Junfeng Zhang, Shaochuan Lin, Zhanming Ou, Tao Luo, Xiaoyi Zeng
Abstract
Transformers have shown promising performance in LLMs due to their outstanding scalability, several studies have investigated the scalability of Transformers for industrial recommendation. They typically rely on a single ranking model to optimize both sparse and dense parameters from scratch, resulting in substantial computational resource consumption and slow convergence. Fortunately, the pre-training models offer an effective solution to the above issues by providing favorable initialization of both sparse and dense parameters for the subsequent ranking. However, they still face two major limitations: (1) Since the input features used in pre-training and ranking are usually inconsistent, directly transferring dense parameters from pre-training to ranking may lead to negative transfer. (2) Multi-epoch training during the ranking process may result in the overfitting of sparse parameters, while freezing the sparse parameters limits their adaptability to the ranking objectives. To this end, we propose a Scaling Transformer for Industrial Recommendation via Transferable Generative Pre-training, termed LazFormer. Specifically, we first present a generative pre-training module to autoregressively generate sequential features, providing favorable initialization of both sparse and dense parameters for the subsequent ranking. To solve the negative transfer of dense parameters, we propose a transferable residual adapter that injects additional ranking-specific features into ranking in a residual manner. Moreover, a request-aware ranking module integrates long-sequence compression, hybrid sparse attention, and a request-aware paradigm to efficiently model users' long sequences. Besides, we further propose an asymmetric multi-epoch training strategy that resets sparse parameters while continuously accumulating dense parameters across epochs, alleviating the overfitting of sparse parameters.