Papers for

cluster operators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Gpu clusters show big gaps in usage efficiency and scheduling

Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring

Abstract: GPU clusters have become essential computing resources for AI and scientific research in academia and industry. However, their high cost and growing demand often result in long queueing delays. While previous monitoring efforts have operated at second- or minute-level frequencies, they primarily informed job-level scheduling for clusters and lacked the detail needed to optimize GPU-level utilization. This paper presents an in-depth study of a university research GPU cluster, analyzing resource usage at 100 ms intervals. We have developed a lightweight monitoring tool that leverages Nvidia's DCGM library to capture fine-grained "compute utilization" metrics within GPUs. Deployed on a cluster with 89 nodes and 319 GPUs, our tool collected data over several months. This paper focuses on a 10-day period with 7,399 jobs submitted by 150 users from 12 departments. Our study reveals three key findings: (1) the GPU cluster exhibits significant underutilization, with over 70% of jobs using less than 20% of GPU memory and 50% averaging below 5% Streaming Multiprocessor (SM) utilization; (2) 80% of jobs have a drastic mismatch between their requested and actual durations, significantly limiting the scheduler's effectiveness; and (3) the multi-GPU jobs we examined show highly periodic communication patterns. These findings suggest there is plenty of room in the middle: substantial utilization gains are available through better job scheduling and GPU multitasking, independent of hardware or application improvements. Our monitoring solution is available as open-source software at https://github.com/als244/gpu_server_monitoring.

Mon 26 OctArtificial IntelligenceDistributed, Parallel, and Cluster Computing
The gist
Gpu clusters are powerful computing systems used for AI and science, but many jobs run on them use only a tiny part of their GPU power or memory. The authors studied a large university gpu cluster closely, measuring how busy the GPUs were every 100 milliseconds. They found most jobs underused the GPUs and often ran for much shorter or longer than planned, making scheduling inefficient. The study also saw repeated patterns in how multiple GPUs communicate during jobs. This means we can make better use of existing gpu clusters just by improving how jobs are assigned and managed, without needing new hardware.
Open → 2610.36329v1

Gpu cluster simulator improves training time predictions with network modeling

Accurate Simulation of Distributed Training Jobs with Network Contention Modeling

Abstract: Trace-driven simulation is widely used to evaluate distributed training (DT) jobs in GPU clusters, but existing simulators either ignore network contention or approximate it with a fixed penalty. This misses how scheduling decisions determine which jobs share server network interfaces and inter-server links, thereby changing networking time during training. As a result, our motivating experiments demonstrate that they incur large errors, reaching up to 73.64% mean absolute percentage error (MAPE) in average job completion time (JCT). This paper introduces MoSim, a GPU-cluster simulator that models DT job execution under dynamic network contention. MoSim combines GPU-free characterization with network contention model: it obtains each job's compute time, networking time, and networking volume without GPUs, then uses the current worker assignment to estimate how shared network interfaces affect each job's iteration time. Our evaluation shows that, compared with existing simulators, MoSim reduces simulation error for average JCT by up to 3.28$\times$, tail (99th-percentile) JCT by up to 7.79$\times$, and makespan by up to 8.48$\times$, while modeling NIC contention factors with only 8.63% error on average. By avoiding real-GPU profiling, MoSim also reduces input construction overhead by 44.6$\times$.

Sun 20 SeptDistributed, Parallel, and Cluster ComputingNetworking and Internet Architecture
The gist
Predicting how long it takes to train AI models on multiple GPUs can be tricky because many simulators ignore slowdowns caused by network sharing. The authors created a simulator called MoSim that includes how jobs compete for network resources, leading to much more accurate predictions of training times. Their tool also avoids using actual GPUs to collect data, making the setup faster and easier. This work helps understand how scheduling affects training speed in GPU clusters.
Open → 2609.23278v1