Training stability challenges and solutions for large recommender systems
Understanding and Mitigating Training Instability in Large-Scale Recommender Systems: A Cascade Perspective
Information RetrievalMachine Learning
Summary
Training big recommendation models that deal with lots of daily user data can run into serious problems where the learning process spirals out of control. The authors studied how this instability happens in a popular video app's recommendation system, identifying a chain reaction that starts with sudden spikes in training signals. They proposed specific fixes targeting each step of this chain to keep training stable. Their experiments showed these methods help maintain reliable and effective recommendations in constantly changing environments.
What this means in practice
- •For machine learning engineers: Improve stability of large-scale recommender system training on platforms with rapidly changing user behavior.
- •For data platform teams: Maintain robust continuous model updates in dynamic data environments by mitigating training divergence at multiple stages.
Authors
Yuening Li, Zeyu Wang, Zhen Zhang, Jiayi Liu, Zhimeng Jiang, Yuji Roh, Weikai Xu, Mingyan Gao, Onkar Dalal, Chuan He, Liang Liu
Abstract
Training large-scale recommender systems on dynamic industrial platforms, such as Short-Form Video (SFV) services with billions of daily interactions, faces critical stability challenges. Rapid distribution shifts in user data and increasing model complexity frequently trigger a divergence cascade, typically initiating with gradient explosion, propagating to weight explosion, and culminating in loss explosion. This G-W-L cascade severely impedes model convergence and performance. This paper provides a systematic analysis of this G-W-L cascade within an industrial SFV recommender context. We then propose and evaluate a structured mitigation framework where interventions are explicitly mapped to counteract specific stages of the cascade: gradient (G) control, weight/activation (W) stabilization, and loss (L) shaping. Our contribution provides a structured framework for understanding and tackling training divergence, offering practical insights for deploying robust, continuously updated recommenders in large-scale, non-stationary settings. We empirically validate the effectiveness of these strategies on a major industrial SFV recommendation system.