Training time efficiency improved for large recommendation systems
Optimizing Effective Training Time for Large-Scale Recommendation Systems
Information Retrieval
Summary
Training very large recommendation systems takes a lot of time on thousands of GPUs, but not all that time actually helps the computer learn from new data. The authors studied why so much time gets wasted and found ways to reduce it by improving how different parts of the system work together. These changes include smarter ways to handle data, reuse computations, and save progress without slowing everything down. After making these improvements, the systems spend much more of their time actually learning, up from about 80% to over 90%.
What this means in practice
- •For machine learning engineers: Increase the proportion of training time spent on actual model learning during large-scale recommendation system training jobs.
- •For data center operations teams: Reduce infrastructure wasted time in GPU clusters by identifying and targeting lifecycle inefficiencies in training workflows.
Authors
Mingming Ding, Ruilin Chen, Yuzhen Huang, Hang Qi, Menglu Yu, San Tan, Damian Reeves, Boris Sarana, Kevin Tang, Satendra Gera, Gagan Jain, Sahil Shah, Vishwa Karia, Fuzail Khan, Yashasvi Makin, Edward Z. Yang, Oguz Ulgen, Jia Chen Ren, Laith Sakka, Mayank Garg, Meet Vadakkanchery, Aici Lin, Wei Sun, Mengjiao Zhou, Shuai Yang, Junqing Zhou, Max Leung, Apoorv Purwar, Musharaf Sultan, John Bocharov, Zhenyu Tang, Vivek Trehan
Abstract
Lifecycle overhead silently consumes accelerator capacity across large-scale recommendation training fleets. Our largest recommendation workloads process tens of billions train- ing examples per day on thousands of GPUs. Before this work, only 50-60% of their end-to-end wall time advanced training on new data. We present a fleet-scale study of this lifecycle overhead and a set of optimizations spanning the full training stack. We use Effective Training Time (ETT%) as an operational framework to instrument lost time, localize it to independently owned infrastructure components, and expose work repeated across job restarts. This analysis guides optimizations like communication elimination and pipeline overlap during trainer initialization; dynamic-shape handling, autotuning pruning, and reusable Py- Torch 2 compilation caches; asynchronous checkpointing; stan- dalone model publishing; and reductions in recovery cost. We evaluate the optimizations on representative models and measure their impacts in our training fleet. ETT% improves on every benchmark, by 15.5% on average, and reaches 85% on our largest workload. Fleet-wide ETT% rose from about 80% to above 90% after deployment.