Papers for

cloud compute schedulers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Safe walltime policies improve supercomputer scheduling efficiency

Evaluating Runtime Prediction and Conservative Walltime Policies in HPC Systems

Abstract: Efficient use of large-scale supercomputing systems depends on accurate runtime estimation and walltime selection. To avoid premature termination, users often overestimate walltime, leading to poor resource utilization and reduced scheduling efficiency. This paper presents a predictor-agnostic framework for safe walltime recommendation in production HPC environments. Rather than proposing a new runtime prediction model, the primary contribution is the SAFE decision layer, which converts predicted runtimes into conservative bounded walltime recommendations suitable for deployment. Runtime prediction is therefore treated as a modular component within the proposed framework. We evaluate the approach using large-scale production job histories from the Fugaku and NREL Eagle supercomputers, comprising approximately 24M and 11M jobs, respectively. Runtime prediction performance is first evaluated using representative predictors, including linear regression, neural networks, and a neural ensemble. The proposed SAFE framework is then evaluated using the NN ensemble predictor. Experimental results indicate that similarity-aware prediction substantially improves runtime estimation for workloads similar to previous executions. When combined with the proposed conservative walltime recommendation policies, these improvements enable walltime gap reductions of approximately 20 - 26% while maintaining job termination rates below 1% across both systems.

Tue 20 OctDistributed, Parallel, and Cluster Computing
The gist
Supercomputers run big tasks that need time limits called walltimes. People often guess these times too long to avoid stopping their jobs early, but that wastes resources. The authors created a framework to safely suggest shorter, more accurate walltimes based on predicted runtimes. Tested on millions of jobs from two supercomputers, their approach cut wasted time by about a quarter while keeping job failures below 1%. This helps supercomputers work better and faster.
Open → 2610.27755v1