Workflow graph structure improves prediction of task resource use
About the Influence of Workflow Topology on Task Intensity Prediction through Graph Learning
Distributed, Parallel, and Cluster ComputingMachine Learning
Summary
Workflows in cloud computing are like maps showing the order of tasks. Predicting how much CPU or memory each task will use helps avoid slowdowns or wasted resources. This paper by the authors shows that not just task details but also the overall shape of this map (the workflow's graph) matters a lot. They tested various methods and found that using the graph's structure helps make much better predictions, especially when combined with models designed for graphs.
What this means in practice
- •For cloud infrastructure teams: Predict task resource needs more accurately by including workflow graph patterns, reducing bottlenecks and resource waste during cloud workflow execution.
- •For data pipeline engineers: Improve scheduling and resource allocation in complex data processing pipelines by modeling task resource intensity with graph-based topology features.
Authors
Max Otto, Haci Ismail Aslan, Joel Witzke, Jonathan Bader, Odej Kao
Abstract
Efficient resource provisioning for large-scale workflows on cloud infrastructures is a critical performance engineering challenge. These workflows are often structured as directed acyclic graphs (DAGs), where under-provisioning can cause critical bottlenecks and over-provisioning leads to unnecessary costs. Accurate, task-level prediction of resource intensity (e.g., CPU load and memory usage) is essential for mitigating these issues. While task-level features are commonly used for prediction, the performance impact of the workflow's overall topological structure is often overlooked or assumed. The central question of our work is: To what extent does what part of the DAG topology influence task-level resource intensity, and what is the most effective way to model this influence? This paper presents a comprehensive benchmark to systematically quantify the impact of graph topology on task intensity prediction. We evaluate and compare a spectrum of modeling approaches. Our findings demonstrate that topology is a critical feature for accurate prediction. Models incorporating important topological information, even through simple handcrafted features, significantly outperform baseline models. We show that graph-native models provide the highest accuracy, achieving low mean absolute errors for both CPU and memory predictions, and can still be combined with simple topological features that they do not learn for better performance.