Hierarchical graph approach improves cloud task scheduling efficiency
HiGFRL: Hierarchical Graph Fusion-Driven Reinforcement Learning for Dependency-Aware Task Scheduling in Heterogeneous Cloud
Machine Learning
Summary
Scheduling tasks in cloud computing is hard because tasks often depend on each other and the computer resources vary. The authors propose HiGFRL, a new method that uses different types of graphs to understand both the task dependencies and the available resources in real-time. This helps the system make smarter choices about where to run each task, reducing waiting times and speeding up the overall work. Tests on real cloud data show this method cuts down the total job time significantly compared to older methods.
What this means in practice
- •For cloud infrastructure teams: Optimize scheduling of complex, dependent tasks on diverse cloud resources to reduce job completion time and improve overall throughput.
- •For data center operators: Improve resource usage and lower task queuing delays in high-load cloud clusters by adopting hierarchical graph-based scheduling models.
Authors
Tiangang Li, Shi Ying, Xiangbo Tian
Abstract
Online scheduling of dependency-aware tasks in heterogeneous cloud clusters is a fundamental yet challenging problem due to the complex interplay between DAG topologies and multi-dimensional resource constraints. While DRL has shown promise, existing GNN-based approaches often struggle to efficiently model high-order topological dependencies and suffer from loose coupling between task and resource states, leading to myopic scheduling decisions. To address these limitations, we propose HiGFRL, a Hierarchical Graph Fusion-Driven Reinforcement Learning framework. HiGFRL constructs a novel three-level state representation comprising a Static Hypergraph, a Dynamic Global Graph, and a Local Bipartite Graph to explicitly model the interplay between task dependencies and real-time cluster dynamics. Specifically, we design a fusion-driven dual-network architecture to optimize RL decision-making, where a Context Fusion Allocator integrates local bipartite matching features with fused global context to execute precise task-to-node allocation, and a Global State Evaluator leverages the global dynamic graph representation to accurately estimate expected long-term cumulative reward. Furthermore, we incorporate a topology-prior-guided hybrid reward mechanism that distills static topological priors into the learning process to accelerate convergence. Extensive experiments using real-world Alibaba cluster traces demonstrate that HiGFRL significantly outperforms heuristics and DRL baselines. Specifically, in challenging large-scale high-load scenarios, HiGFRL reduces the Makespan by up to 32.55%, and optimizes the average task flow time and average task wait time by 13.58% and 13.79%, respectively. Experimental results confirm that HiGFRL not only significantly improves cluster throughput but also ensures superior QoS by substantially reducing queuing delays. Code Release:https://github.com/igeng/HiGFRL.