Mathematical model improves scheduling for large scientific workflows

Mathematical Modeling of a Cognitive Continuum Digital Shadow for Large-Scale, Cross-Facility Workflows

Performance

Summary

Large scientific workflows often run across many computing centers and instruments, making it hard to plan and manage resources efficiently. The authors present a mathematical approach called the Cognitive Continuum Digital Shadow (CCDS) that helps users and operators understand and predict trade-offs like cost, time, and energy before starting jobs. This approach uses advanced optimization to handle uncertain availability of resources, allowing better decisions on where and when to run parts of a workflow. They demonstrated this method on a genome analysis example using diverse computing resources.

What this means in practice

  • For hpc system operators: Plan and optimize cross-facility scientific workflows by quantifying cost, time, and energy trade-offs under uncertain resources before execution.
  • For data center managers: Schedule large computing jobs involving diverse facilities more efficiently with mathematical tools that anticipate variable resource availability.

Authors

Mark Asch, Marius Garénaux Gruau, François Bodin

Abstract

We present the mathematical foundations of a \emph{Cognitive Continuum Digital Shadow} (CCDS), a decision-support layer between users and the cross-facility infrastructure---instruments, networks, data stores and compute centers---of exascale and post-exascale scientific workflows. The CCDS couples a state-space representation of the continuum with multistage stochastic programming, so that deployment scenarios can be explored and optimized \emph{before} jobs are launched. This allows operators and users to quantify the cost, makespan and energy trade-offs of a workflow under uncertain resource availability, and hedge their decisions accordingly. We formulate the underlying optimization as a multimode, resource-constrained, stochastic supply-chain network design problem and demonstrate it on a realistic genomics workflow scheduled across heterogeneous HPC and data-center resources. This is the first of three papers; the second treats the underlying software architecture and the third reports large-scale use-cases.