Ineffective gpu use in clusters cut by scheduling based on readiness prediction
Ineffective Occupancy in Heterogeneous GPU Clusters: Prediction, Contribution Analysis, and Readiness-Aware Scheduling
Distributed, Parallel, and Cluster Computing
Summary
In GPU clusters used for running jobs, GPUs can be wasted when jobs are assigned but never actually ready to run. The authors call this problem ineffective occupancy and found it happens a lot in a big Alibaba cluster. They created a method called Readiness-Aware Scheduling (RAS) that predicts whether a job will be ready based on nine features before starting it. By using RAS, much less GPU time is wasted by jobs that never run, making the cluster work better without losing capacity. Their method works well by combining these features rather than using simple rules about one characteristic alone.
What this means in practice
- •For gpu cluster operators: Improve job scheduling by predicting readiness at admission to reduce wasted gpu time in heterogeneous clusters.
- •For cloud infrastructure teams: Enhance resource efficiency of gpu services by integrating readiness-aware admission policies for heterogeneous gpu hardware.
Tested on one dataset.
Authors
Tao Zhang, Bin Liao
Abstract
In heterogeneous GPU clusters, GPUs are costly and scarce, yet placing a job on a machine does not mean that the job becomes ready to run. A large amount of allocated capacity can remain occupied for long periods without producing useful computation, further aggravating the waste of compute resources. On the Alibaba GPU cluster trace we find that, even when GPU capacity is sufficient and placement is complete, 52.70% of execution records never became ready. We term this phenomenon ineffective occupancy. Building on this diagnosis, and keeping the placement policy fixed, we propose Readiness-Aware Scheduling (RAS): it uses only nine attributes observable at admission time to predict readiness probability and reorders admission accordingly. Under a five-fold split grouped by machine, the gradient-boosted model reaches an ROC-AUC of 0.8187. A feature-group contribution analysis shows that the model's score is driven mainly by job profile, duration, and GPU model, with the request-size group contributing least, followed by the scheduling and concurrency group. In GPU terms, RAS reduces the ineffective-occupancy ratio (the share of cluster GPUs held under ineffective occupancy) from 43.64% to 27.15% and raises the effective coverage of large requests (eight GPUs or more) from 46.11% to 63.18%, without sacrificing capacity utilization. Relative to the upper bound obtained by ordering with ground-truth labels, RAS captures 65.1% of the attainable improvement, well above the interpretable rules built only from the contribution-analysis conclusions; the gain therefore comes mainly from predicting with all nine attributes jointly rather than from combining rules built on any single attribute group. These results indicate that readiness status is a measurable, predictable, and optimizable dimension of admission decisions in heterogeneous GPU clusters.