Papers for

hpc cluster administrators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

SafeWall improves job runtime estimates in high-performance computing

SafeWall: A Risk-Aware Job Walltime Recommendation for HPC Scheduling

Abstract: In high-performance computing (HPC), users must submit resource requirements for their jobs, with user-supplied walltime being a critical parameter. However, these estimates are notoriously inaccurate, as users tend to overestimate walltime to prevent the scheduler from terminating their jobs prematurely. While existing work on learning-based walltime prediction primarily focuses on maximizing prediction accuracy, such predictions are inherently risky for walltime recommendations. Specifically, underestimation leads to job termination before completion, resulting in lost progress and wasted resources. In this study, we present SafeWall, a novel recommendation method that adjusts walltime predictions by incorporating an explicit, nonnegative safety buffer to mitigate underestimation risk. The core of our approach is a risk-aware buffer that dynamically calibrates safety margins based on individual user's historical behavior. When user data is limited, the system employs statistical shrinkage to align these margins with a global baseline, ensuring robustness. This yields a lightweight mechanism that optimizes the trade-off between prediction accuracy and termination risk. SafeWall is modular and applicable to various learning-based prediction models. We evaluate SafeWall using five public workload traces, including the full-lifetime job log from the 4,392-node Theta machine at the Argonne Leadership Computing Facility. Across all traces, SafeWall substantially reduces underestimation risk compared to the unbuffered base predictors, while preserving competitive accuracy. The source code is publicly available at https://github.com/SPEAR-UIC/JSSPP26-SafeWall.

Mon 19 OctDistributed, Parallel, and Cluster ComputingMachine Learning
The gist
In high-performance computing, users must guess how long their jobs will run, but they often overestimate to avoid jobs ending too soon. The authors present SafeWall, a way to adjust these guesses by adding a safety buffer based on each user's past behavior. This approach reduces the chance that jobs get cut off early without losing prediction accuracy. Their method works with many prediction models and was tested on real job data from large computing centers.
Open → 2610.26577v1