Real-time scheduling boosts data processing on exascale computers
Real-time Optimization of Simulation and Data Processing Pipelines for Experiments on Exascale Computing Platforms
Distributed, Parallel, and Cluster Computing
Summary
Big scientific experiments create huge amounts of data that need very powerful computers to analyze quickly. The authors developed a way to better organize and time tasks on these supercomputers to use their power more efficiently. They tested this on a simulation from a particle physics experiment, showing how to pick the best way to run tasks and allocate resources. Their methods help manage complex workflows so important research can happen faster and use less computing power.
What this means in practice
- •For scientific computing teams: Optimize task scheduling to improve efficiency in large-scale scientific data processing workflows on exascale platforms.
- •For cloud computing operators: Implement multi-stage scheduling methods to enhance resource utilization for complex simulation pipelines in cloud or HPC environments.
Authors
Thomas Wester, Harikrishna Tummalapalli, Christine M. Simpson
Abstract
Current and next-generation large-scale experiments, especially in high-energy physics and cosmology, involve increasing volumes of data that require exascale computing power to analyze and model. Exascale machines present challenges for workflow scaling and efficient resource usage. We propose a multi-stage task scheduling approach for optimizing resource usage for these workflows. We demonstrate task scheduling for an example Monte Carlo simulation pipeline from the Short-Baseline Near Detector (SBND), and present theoretical and numerical tools for modeling task scheduling in these workflows that can be used to identify optimal scheduling strategies and choose resources for a given problem size.