Local method predicts gpu latency for low-end graphics cards
Local Zero-Shot Latency Prediction for Low-End GPUs
Distributed, Parallel, and Cluster ComputingPerformance
Summary
Measuring how fast GPU hardware runs machine learning tasks can be expensive, especially for many lower-end GPUs. The authors show that you can adjust settings on a local, more powerful GPU to mimic the slower performance of less powerful GPUs. This lets them predict latency on low-end GPUs without needing to physically test on them. Their approach makes latency predictions more accurate and also preserves the order of speeds among different GPUs.
What this means in practice
- •For machine learning engineers: Estimate the runtime of vision models on lower-end GPUs using a single configured local GPU without extensive hardware testing.
- •For mobile app developers: Predict how machine learning features will perform on a wide range of consumer GPUs, especially low-end devices, before deployment.
Authors
Arno Geimer, Beltran Fiz, Dimitris Stripelis
Abstract
Quantifying execution latency is an important consideration across Machine Learning. Existing latency-prediction approaches typically train ML models on traces collected from workload execution on target hardware. Collection of representative traces across many consumer GPUs is expensive both in organizing the devices and in manpower, causing existing work to mainly focus on a limited set of high-end GPUs available in cloud computing. To offset this cost, we show that a local host GPU can be configured to reproduce the execution performance of less powerful target GPUs. Through selectively modifying device parameters, we obtain locally measured latencies that closely approximate those observed on the target hardware. We reformulate the latency-prediction task: instead of fitting a prediction model to a dataset of execution traces, we identify local configurations that best reproduce target devices' performance, enabling zero-shot prediction. We evaluate our approach against real measurements from 60 computer-vision models executed on 21 consumer GPU models. Our method achieves a mean absolute percentage error (MAPE) of 14.21%, compared with more than 25% for comparable baselines, reducing relative prediction error by 43.6%. Predicted and measured latencies exhibit an average Spearman correlation of 0.961 across training tasks, demonstrating that the method also accurately preserves performance rankings across devices.