CQTwin toolkit simulates HPC schedules for better workload decisions
CQTwin: A Digital-Twin Toolkit for Operational Analytics of HPC Schedulers
Distributed, Parallel, and Cluster ComputingPerformance
Summary
Large computing centers have many records of how computing jobs were scheduled but lack easy tools to learn from this data. The authors present CQTwin, a software tool that can replay past workloads and simulate how different scheduling policies might change outcomes. It works both offline with stored data and online with live job events, helping operators compare policies and plan better decisions. They tested CQTwin with real-world data from a major research facility over several years.
What this means in practice
- •For hpc system operators: Evaluate and compare scheduling policies on real historical workload data to improve job throughput and resource use.
- •For cloud infrastructure teams: Test alternative scheduling strategies live on queued jobs to anticipate resource demands and optimize cloud workload placement.
Authors
Kanglin Xu, Yash Kurkure, Yihe Zhang, Michael E. Papka, Zhiling Lan
Abstract
HPC facilities collect years of job scheduler records, yet much of this operational data remains underused because accessible tools for turning it into actionable insight are scarce. We present CQTwin, an open-source digital-twin toolkit for operational analytics of HPC scheduling data. CQTwin addresses this gap through two complementary modes: offline trace replay and online event-driven simulation. Built on the Python-based CQSim discrete-event simulator, CQTwin combines a fidelity-validated, high-performance C++ engine with an interactive browser-based dashboard and a Snapshot Lab, enabling facility staff, operators, and researchers to characterize workloads, compare scheduling policies, and explore what-if scenarios. As an offline twin, CQTwin replays archived workload traces under alternative policies to show how the same workload would have unfolded under different scheduling decisions. As an online twin, it consumes job events as they arrive and projects how alternative policies would process the currently queued jobs, providing guidance for real-time decision-making. The dashboard supports the complete analysis workflow, including workload characterization, experiment configuration and monitoring, and multidimensional comparison of simulation results, allowing users to determine visually which policies best suit a workload and what trade-offs they entail. We validate CQTwin using production traces from the Argonne Leadership Computing Facility spanning 2016-2023.