Cognitive system helps plan large scientific workflows more efficiently
Mathematical Modeling of a Cognitive Continuum Digital Shadow for Large-Scale, Cross-Facility Workflows
Performance
Summary
Running big scientific experiments often involves many computers, storage, and networks working together. The paper shows a way to help users decide how to run these jobs by predicting costs, time, and energy before starting. The authors use math to model the whole system and handle uncertainties like resources might be busy or slow. They tested their approach on a genetics data workflow across different computing centers. This work is part one of three, with future papers focusing on software design and real-world examples.
Cognitive continuumDigital shadowMultistage stochastic programmingState-space representationExascale computingScientific workflowsResource allocationSupply-chain network designHigh-performance computingGenomics workflow
Authors
Mark Asch, Marius Garénaux Gruau, François Bodin
Abstract
We present the mathematical foundations of a \emph{Cognitive Continuum Digital Shadow} (CCDS), a decision-support layer between users and the cross-facility infrastructure---instruments, networks, data stores and compute centers---of exascale and post-exascale scientific workflows. The CCDS couples a state-space representation of the continuum with multistage stochastic programming, so that deployment scenarios can be explored and optimized \emph{before} jobs are launched. This allows operators and users to quantify the cost, makespan and energy trade-offs of a workflow under uncertain resource availability, and hedge their decisions accordingly. We formulate the underlying optimization as a multimode, resource-constrained, stochastic supply-chain network design problem and demonstrate it on a realistic genomics workflow scheduled across heterogeneous HPC and data-center resources. This is the first of three papers; the second treats the underlying software architecture and the third reports large-scale use-cases.