Parameter inheritance improves scaling and efficiency of recommendation models
Inherit4Rec: Parameter Inheritance for Efficient Scaling of Recommendation Models
Information Retrieval
Summary
Recommender systems often need bigger models to improve their suggestions but training large models from scratch takes a lot of time and computing power. The authors propose a method called Inherit4Rec that helps grow models or make them sparser by reusing parameters from smaller models, so training is faster and requires less data. Their approach also handles changing recommendation data better than previous methods. Tests on public and industrial datasets show Inherit4Rec works well to keep improving recommendations while saving computation.
What this means in practice
- •For recommendation engineering teams: Scale recommendation models more efficiently by inheriting parameters, reducing training time and resources while maintaining or improving prediction quality.
- •For machine learning infrastructure teams: Deploy sparse recommendation models with balanced computation loads using parameter inheritance to cut serving costs and meet strict latency budgets.
- •For advertising platform developers: Use parameter inheritance to quickly expand or compress recommendation models, enabling faster adaptation to changing user data and campaign needs.$Commercial implications: Enables scalable, efficient recommendation model updates that improve ad targeting platforms, potentially sold as model optimization tools or services.
Authors
Ruihao Zhang, Bo Chen, Xiao Wang, Jinlong Jiao, Tijian Hu, Qinglin Jia, Xiuqiang He, Xiangyu Zhao, Chaoyi Ma, Ruiming Tang, Wenwu Ou
Abstract
Scaling model capacity has emerged as an effective approach to overcoming performance bottlenecks in industrial recommender systems. However, repeatedly training larger dense models from scratch demands substantial data and time, while their growing computation conflicts with the strict serving budgets of industrial systems. Parameter inheritance provides a promising route for both dense model growth and sparse conversion, yet existing methods are primarily designed for static corpora and can suffer sharp performance drops under dynamically evolving recommendation data. To address these challenges, we propose Inherit4Rec, a parameter-inheritance framework that supports both Dense-to-Dense (D2D) growth and Dense-to-Sparse (D2S) conversion. Inherit4Rec-D2D combines hybrid growth with asymmetric training to preserve the forward function at expansion and maintain update continuity. Inherit4Rec-D2S constructs SMoE networks through co-activation-aware partitioning and a load-balancing loss, preserving dense-model capabilities while promoting balanced expert activation. Experiments on KuaiRand-1K and an industrial short-video recommendation dataset show that both transformations consistently outperform the evaluated inheritance baselines across all prediction objectives. These results demonstrate the effectiveness of Inherit4Rec for continual capacity expansion and computation-efficient sparse conversion in industrial recommender systems.