Gpu clusters show big gaps in usage efficiency and scheduling
Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring
Artificial IntelligenceDistributed, Parallel, and Cluster Computing
Summary
Gpu clusters are powerful computing systems used for AI and science, but many jobs run on them use only a tiny part of their GPU power or memory. The authors studied a large university gpu cluster closely, measuring how busy the GPUs were every 100 milliseconds. They found most jobs underused the GPUs and often ran for much shorter or longer than planned, making scheduling inefficient. The study also saw repeated patterns in how multiple GPUs communicate during jobs. This means we can make better use of existing gpu clusters just by improving how jobs are assigned and managed, without needing new hardware.
What this means in practice
- •For cluster operators: Improve scheduling policies by using fine-grained data to reduce GPU idle times and job duration mismatches.
- •For cloud service providers: Optimize multi-GPU usage and resource allocation in data centers by leveraging detailed GPU utilization and communication pattern insights.
Authors
Andrew Sheinberg, Samyak Gupta, Kai Li
Abstract
GPU clusters have become essential computing resources for AI and scientific research in academia and industry. However, their high cost and growing demand often result in long queueing delays. While previous monitoring efforts have operated at second- or minute-level frequencies, they primarily informed job-level scheduling for clusters and lacked the detail needed to optimize GPU-level utilization. This paper presents an in-depth study of a university research GPU cluster, analyzing resource usage at 100 ms intervals. We have developed a lightweight monitoring tool that leverages Nvidia's DCGM library to capture fine-grained "compute utilization" metrics within GPUs. Deployed on a cluster with 89 nodes and 319 GPUs, our tool collected data over several months. This paper focuses on a 10-day period with 7,399 jobs submitted by 150 users from 12 departments. Our study reveals three key findings: (1) the GPU cluster exhibits significant underutilization, with over 70% of jobs using less than 20% of GPU memory and 50% averaging below 5% Streaming Multiprocessor (SM) utilization; (2) 80% of jobs have a drastic mismatch between their requested and actual durations, significantly limiting the scheduler's effectiveness; and (3) the multi-GPU jobs we examined show highly periodic communication patterns. These findings suggest there is plenty of room in the middle: substantial utilization gains are available through better job scheduling and GPU multitasking, independent of hardware or application improvements. Our monitoring solution is available as open-source software at https://github.com/als244/gpu_server_monitoring.